2024-09-01T02:46:48.081 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-01T02:46:48.093 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-01T02:46:48.208 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi/7884281 branch: wip-bharath2-testing-2024-08-31-2129 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} email: skanta@redhat.com first_in_suite: false job_id: '7884281' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath2-testing-2024-08-31-2129 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5028887063e8eb518c95117479aa913eaa12ac8e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5028887063e8eb518c95117479aa913eaa12ac8e workunit: branch: wip-bharath2-testing-2024-08-31-2129 sha1: 5028887063e8eb518c95117479aa913eaa12ac8e owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 881 sha1: 5028887063e8eb518c95117479aa913eaa12ac8e sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath2-testing-2024-08-31-2129 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5028887063e8eb518c95117479aa913eaa12ac8e targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDg/n0nCxSD7vyStkPFdULR4GICz2lUnMUJ8kKfnz0T2QRoz9QZWyCqFNFN8zJWt6Dc9t7ljihjehqyrYSc/7w= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-31_23:59:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-01T02:46:48.208 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa; will attempt to use it 2024-09-01T02:46:48.209 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks 2024-09-01T02:46:48.209 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-01T02:46:48.211 INFO:teuthology.task.internal:Checking packages... 2024-09-01T02:46:48.231 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5028887063e8eb518c95117479aa913eaa12ac8e' 2024-09-01T02:46:48.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-01T02:46:48.232 INFO:teuthology.packaging:ref: None 2024-09-01T02:46:48.232 INFO:teuthology.packaging:tag: None 2024-09-01T02:46:48.232 INFO:teuthology.packaging:branch: wip-bharath2-testing-2024-08-31-2129 2024-09-01T02:46:48.232 INFO:teuthology.packaging:sha1: 5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T02:46:48.232 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath2-testing-2024-08-31-2129 2024-09-01T02:46:48.372 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4640.g50288870 2024-09-01T02:46:48.374 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-01T02:46:48.382 INFO:teuthology.task.internal:no buildpackages task found 2024-09-01T02:46:48.382 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-01T02:46:48.605 INFO:teuthology.task.internal:Saving configuration 2024-09-01T02:46:48.613 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-01T02:46:48.633 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-01T02:46:48.659 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi/7884281', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-01 02:41:50.883527', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDg/n0nCxSD7vyStkPFdULR4GICz2lUnMUJ8kKfnz0T2QRoz9QZWyCqFNFN8zJWt6Dc9t7ljihjehqyrYSc/7w='} 2024-09-01T02:46:48.659 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-01T02:46:48.668 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-01T02:46:48.668 INFO:teuthology.run_tasks:Running task console_log... 2024-09-01T02:46:48.717 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f66bf223be0>, signals=[15]) 2024-09-01T02:46:48.717 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-01T02:46:48.724 INFO:teuthology.task.internal:Opening connections... 2024-09-01T02:46:48.724 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-09-01T02:46:48.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-01T02:46:48.800 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-01T02:46:48.808 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-09-01T02:46:48.882 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-09-01T02:46:48.883 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-09-01T02:46:48.941 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-09-01T02:46:48.942 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-01T02:46:48.942 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-09-01T02:46:48.942 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-01T02:46:48.942 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-01T02:46:48.942 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-01T02:46:48.942 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-09-01T02:46:48.965 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-01T02:46:48.973 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-01T02:46:49.220 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-01T02:46:49.220 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-09-01T02:46:49.238 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-01T02:46:49.246 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-01T02:46:49.246 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-09-01T02:46:49.305 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-01T02:46:49.338 INFO:teuthology.run_tasks:Running task kernel... 2024-09-01T02:46:49.354 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-01T02:46:49.354 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-01T02:46:49.354 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-01T02:46:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:46:49.372 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-01T02:46:49.430 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-503.el9.x86_64 2024-09-01T02:46:49.430 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-503.el9.x86_64 2024-09-01T02:46:49.430 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-09-01T02:46:51.773 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-01T02:46:54.832 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-01T02:46:59.858 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.6 MB 00:01 2024-09-01T02:47:01.317 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 18 kB 00:00 2024-09-01T02:47:02.754 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 17 MB/s | 22 MB 00:01 2024-09-01T02:47:08.210 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-01T02:47:10.233 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-01T02:47:10.233 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-01T02:47:10.318 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:47:10.319 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:47:10.319 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:47:10.439 DEBUG:teuthology.orchestra.run.smithi079:> echo no | sudo yum reinstall kernel || true 2024-09-01T02:47:10.973 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:02 ago on Sun 01 Sep 2024 02:47:08 AM UTC. 2024-09-01T02:47:11.117 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:47:11.123 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:47:11.124 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 1.9 M 2024-09-01T02:47:11.124 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-09-01T02:47:11.124 INFO:teuthology.orchestra.run.smithi079.stderr:Operation aborted. 2024-09-01T02:47:11.180 INFO:teuthology.orchestra.run.smithi079.stdout:Is this ok [y/N]: 2024-09-01T02:47:11.180 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum reinstall -y kernel || true 2024-09-01T02:47:11.713 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:03 ago on Sun 01 Sep 2024 02:47:08 AM UTC. 2024-09-01T02:47:11.841 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:47:11.842 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:47:11.842 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-01T02:47:11.842 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:47:11.842 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalling: 2024-09-01T02:47:11.842 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 1.9 M 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 0 2024-09-01T02:47:11.843 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:47:12.553 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-503.el9.x86_64.rpm 4.1 MB/s | 1.9 MB 00:00 2024-09-01T02:47:12.554 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:47:12.554 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2024-09-01T02:47:12.583 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:47:12.836 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:47:12.836 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:47:12.867 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:47:12.867 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:47:13.018 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:47:13.131 INFO:teuthology.orchestra.run.smithi079.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-01T02:47:13.131 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-01T02:47:13.883 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-01T02:47:13.883 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-01T02:47:14.610 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-01T02:47:14.611 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:47:14.611 INFO:teuthology.orchestra.run.smithi079.stdout:Reinstalled: 2024-09-01T02:47:14.611 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-01T02:47:14.611 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:47:14.611 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:47:14.816 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-09-01T02:47:14.875 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-01T02:47:14.875 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-01T02:47:14.876 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-01T02:47:14.876 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-01T02:47:14.876 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-01T02:47:14.907 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-09-01T02:47:14.930 DEBUG:teuthology.parallel:result is None 2024-09-01T02:47:14.931 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-01T02:47:14.938 INFO:teuthology.task.internal:Creating test directory... 2024-09-01T02:47:14.938 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-01T02:47:14.986 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-01T02:47:14.994 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-01T02:47:15.019 INFO:teuthology.task.internal:Creating archive directory... 2024-09-01T02:47:15.020 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-01T02:47:15.053 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-01T02:47:15.060 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-01T02:47:15.060 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-01T02:47:15.139 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:47:15.148 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:47:15.150 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-01T02:47:15.157 INFO:teuthology.task.internal:Configuring sudo... 2024-09-01T02:47:15.157 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-01T02:47:15.182 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-01T02:47:15.193 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-01T02:47:15.193 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-01T02:47:15.238 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-01T02:47:15.323 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-01T02:47:15.372 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-01T02:47:15.453 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-01T02:47:15.494 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:47:15.494 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-01T02:47:15.559 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-09-01T02:47:15.643 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-01T02:47:16.083 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-01T02:47:16.091 INFO:teuthology.task.internal:Starting timer... 2024-09-01T02:47:16.091 INFO:teuthology.run_tasks:Running task pcp... 2024-09-01T02:47:16.132 INFO:teuthology.run_tasks:Running task selinux... 2024-09-01T02:47:16.163 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-09-01T02:47:16.204 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-09-01T02:47:16.206 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-01T02:47:16.206 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-09-01T02:47:16.276 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-09-01T02:47:16.277 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive'} 2024-09-01T02:47:16.277 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-01T02:47:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:47:16.345 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-01T02:47:16.345 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-09-01T02:47:16.419 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-01T02:47:16.428 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-01T02:47:16.431 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-01T02:47:16.569 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-01T02:47:16.584 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-01T02:47:16.585 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-01T02:48:59.146 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com')] 2024-09-01T02:48:59.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-01T02:48:59.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-01T02:48:59.225 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-01T02:48:59.304 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-01T02:48:59.305 INFO:teuthology.run_tasks:Running task clock... 2024-09-01T02:48:59.316 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-01T02:48:59.317 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-01T02:48:59.317 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-01T02:48:59.373 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-01T02:48:59.388 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-01T02:48:59.412 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-09-01T02:48:59.422 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-09-01T02:48:59.438 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-01T02:48:59.452 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-01T02:48:59.512 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-09-01T02:48:59.515 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-01T02:48:59.515 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-09-01T02:48:59.515 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T02:48:59.515 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T02:48:59.515 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T02:48:59.515 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T02:48:59.516 INFO:teuthology.run_tasks:Running task install... 2024-09-01T02:48:59.524 DEBUG:teuthology.task.install:project ceph 2024-09-01T02:48:59.524 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5028887063e8eb518c95117479aa913eaa12ac8e'}} 2024-09-01T02:48:59.524 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5028887063e8eb518c95117479aa913eaa12ac8e'} 2024-09-01T02:48:59.525 INFO:teuthology.task.install:Using flavor: default 2024-09-01T02:48:59.531 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-01T02:48:59.531 INFO:teuthology.task.install:extra packages: [] 2024-09-01T02:48:59.532 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5028887063e8eb518c95117479aa913eaa12ac8e', 'tag': None, 'wait_for_package': False} 2024-09-01T02:48:59.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T02:48:59.677 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/ 2024-09-01T02:48:59.678 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4640.g50288870 2024-09-01T02:48:59.790 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath2-testing-2024-08-31-2129/5028887063e8eb518c95117479aa913eaa12ac8e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-01T02:48:59.790 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:48:59.790 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-01T02:48:59.817 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-01T02:48:59.817 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5028887063e8eb518c95117479aa913eaa12ac8e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-01T02:48:59.883 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-01T02:48:59.959 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-01T02:49:00.023 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-09-01T02:49:00.025 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-09-01T02:49:00.252 INFO:teuthology.orchestra.run.smithi079.stdout:45 files removed 2024-09-01T02:49:00.279 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-09-01T02:49:01.119 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 148 kB/s | 84 kB 00:00 2024-09-01T02:49:01.411 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-01T02:49:01.687 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-01T02:49:04.620 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-09-01T02:49:10.112 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-01T02:49:15.737 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.6 MB 00:01 2024-09-01T02:49:17.713 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 18 kB 00:00 2024-09-01T02:49:22.079 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 22 MB 00:04 2024-09-01T02:49:27.456 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-01T02:49:29.791 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:49:29.793 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:19.3.0-4640.g50288870.el9 ceph 11 M 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:19.3.0-4640.g50288870.el9 ceph 3.5 M 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:19.3.0-4640.g50288870.el9 ceph 3.2 M 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:19.3.0-4640.g50288870.el9 ceph 5.5 M 2024-09-01T02:49:29.794 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:19.3.0-4640.g50288870.el9 ceph 22 M 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:19.3.0-4640.g50288870.el9 ceph 25 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:19.3.0-4640.g50288870.el9 ceph 755 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:19.3.0-4640.g50288870.el9 ceph 508 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:19.3.0-4640.g50288870.el9 ceph 5.5 M 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-01T02:49:29.795 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:19.3.0-4640.g50288870.el9 ceph 45 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:19.3.0-4640.g50288870.el9 ceph 143 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:19.3.0-4640.g50288870.el9 ceph 162 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:19.3.0-4640.g50288870.el9 ceph 321 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:19.3.0-4640.g50288870.el9 ceph 306 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:19.3.0-4640.g50288870.el9 ceph 99 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-01T02:49:29.796 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout:Install 30 Packages 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:29.797 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 62 M 2024-09-01T02:49:29.798 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:49:31.165 INFO:teuthology.orchestra.run.smithi079.stdout:(1/32): ceph-base-19.3.0-4640.g50288870.el9.x86 8.1 MB/s | 5.5 MB 00:00 2024-09-01T02:49:31.307 INFO:teuthology.orchestra.run.smithi079.stdout:(2/32): ceph-radosgw-19.3.0-4640.g50288870.el9. 13 MB/s | 11 MB 00:00 2024-09-01T02:49:31.332 INFO:teuthology.orchestra.run.smithi079.stdout:(3/32): ceph-selinux-19.3.0-4640.g50288870.el9. 151 kB/s | 25 kB 00:00 2024-09-01T02:49:31.374 INFO:teuthology.orchestra.run.smithi079.stdout:(4/32): libcephfs2-19.3.0-4640.g50288870.el9.x8 11 MB/s | 755 kB 00:00 2024-09-01T02:49:31.407 INFO:teuthology.orchestra.run.smithi079.stdout:(5/32): libradosstriper1-19.3.0-4640.g50288870. 6.6 MB/s | 508 kB 00:00 2024-09-01T02:49:31.466 INFO:teuthology.orchestra.run.smithi079.stdout:(6/32): python3-ceph-argparse-19.3.0-4640.g5028 772 kB/s | 45 kB 00:00 2024-09-01T02:49:31.558 INFO:teuthology.orchestra.run.smithi079.stdout:(7/32): python3-ceph-common-19.3.0-4640.g502888 1.5 MB/s | 143 kB 00:00 2024-09-01T02:49:31.633 INFO:teuthology.orchestra.run.smithi079.stdout:(8/32): librgw2-19.3.0-4640.g50288870.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-01T02:49:31.675 INFO:teuthology.orchestra.run.smithi079.stdout:(9/32): python3-cephfs-19.3.0-4640.g50288870.el 1.4 MB/s | 162 kB 00:00 2024-09-01T02:49:31.709 INFO:teuthology.orchestra.run.smithi079.stdout:(10/32): python3-rados-19.3.0-4640.g50288870.el 4.2 MB/s | 321 kB 00:00 2024-09-01T02:49:31.734 INFO:teuthology.orchestra.run.smithi079.stdout:(11/32): python3-rbd-19.3.0-4640.g50288870.el9. 5.1 MB/s | 306 kB 00:00 2024-09-01T02:49:31.760 INFO:teuthology.orchestra.run.smithi079.stdout:(12/32): python3-rgw-19.3.0-4640.g50288870.el9. 1.9 MB/s | 99 kB 00:00 2024-09-01T02:49:31.985 INFO:teuthology.orchestra.run.smithi079.stdout:(13/32): ceph-common-19.3.0-4640.g50288870.el9. 14 MB/s | 22 MB 00:01 2024-09-01T02:49:32.043 INFO:teuthology.orchestra.run.smithi079.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-01T02:49:32.085 INFO:teuthology.orchestra.run.smithi079.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-09-01T02:49:32.111 INFO:teuthology.orchestra.run.smithi079.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 456 kB/s | 160 kB 00:00 2024-09-01T02:49:32.144 INFO:teuthology.orchestra.run.smithi079.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-01T02:49:32.170 INFO:teuthology.orchestra.run.smithi079.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 721 kB/s | 42 kB 00:00 2024-09-01T02:49:32.203 INFO:teuthology.orchestra.run.smithi079.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-01T02:49:32.229 INFO:teuthology.orchestra.run.smithi079.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-01T02:49:32.254 INFO:teuthology.orchestra.run.smithi079.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 429 kB/s | 36 kB 00:00 2024-09-01T02:49:32.279 INFO:teuthology.orchestra.run.smithi079.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-09-01T02:49:32.346 INFO:teuthology.orchestra.run.smithi079.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-01T02:49:32.380 INFO:teuthology.orchestra.run.smithi079.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-09-01T02:49:32.405 INFO:teuthology.orchestra.run.smithi079.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 236 kB/s | 26 kB 00:00 2024-09-01T02:49:32.430 INFO:teuthology.orchestra.run.smithi079.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-01T02:49:32.464 INFO:teuthology.orchestra.run.smithi079.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-01T02:49:32.523 INFO:teuthology.orchestra.run.smithi079.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-01T02:49:32.581 INFO:teuthology.orchestra.run.smithi079.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-01T02:49:32.690 INFO:teuthology.orchestra.run.smithi079.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-01T02:49:32.773 INFO:teuthology.orchestra.run.smithi079.stdout:(31/32): librados2-19.3.0-4640.g50288870.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-01T02:49:32.849 INFO:teuthology.orchestra.run.smithi079.stdout:(32/32): librbd1-19.3.0-4640.g50288870.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-01T02:49:32.856 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:49:32.856 INFO:teuthology.orchestra.run.smithi079.stdout:Total 20 MB/s | 62 MB 00:03 2024-09-01T02:49:32.955 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:49:32.991 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:49:32.991 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:49:33.470 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:49:33.470 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:49:34.942 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:49:35.032 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-01T02:49:35.070 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-01T02:49:35.095 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-01T02:49:35.126 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-01T02:49:35.152 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-01T02:49:35.418 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-01T02:49:35.599 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-01T02:49:36.001 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-01T02:49:36.064 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:19.3.0-4640.g50288870.el9.x86_64 7/34 2024-09-01T02:49:36.156 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:19.3.0-4640.g50288870.el9.x86_64 7/34 2024-09-01T02:49:36.246 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 8/34 2024-09-01T02:49:36.282 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 9/34 2024-09-01T02:49:36.336 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 9/34 2024-09-01T02:49:36.365 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-01T02:49:36.409 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:19.3.0-4640.g50288870.el9. 11/34 2024-09-01T02:49:36.495 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 12/34 2024-09-01T02:49:36.526 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_6 13/34 2024-09-01T02:49:36.582 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_6 13/34 2024-09-01T02:49:36.616 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-01T02:49:36.726 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-01T02:49:36.759 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-01T02:49:36.799 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-01T02:49:36.824 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-01T02:49:36.853 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-01T02:49:36.894 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-01T02:49:37.213 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-01T02:49:37.259 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:19.3.0-4640.g50288870.el9.x86_64 22/34 2024-09-01T02:49:37.350 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:19.3.0-4640.g50288870.el9.x86_64 22/34 2024-09-01T02:49:37.380 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 23/34 2024-09-01T02:49:37.418 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-01T02:49:37.908 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-01T02:49:37.939 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:19.3.0-4640.g50288870.el9.x86_64 26/34 2024-09-01T02:49:38.010 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:19.3.0-4640.g50288870.el9.x86_64 26/34 2024-09-01T02:49:38.088 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 27/34 2024-09-01T02:49:38.125 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:19.3.0-4640.g50288870.el9.x8 28/34 2024-09-01T02:49:40.410 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 29/34 2024-09-01T02:49:40.439 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 29/34 2024-09-01T02:49:40.993 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 29/34 2024-09-01T02:49:41.043 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 30/34 2024-09-01T02:49:41.128 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 30/34 2024-09-01T02:49:41.128 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-01T02:49:41.128 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-01T02:49:41.129 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:41.161 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 31/34 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 31/34 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-09-01T02:49:51.067 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:52.029 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 32/34 2024-09-01T02:49:52.055 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 32/34 2024-09-01T02:49:52.055 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:49:52.055 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-01T02:49:52.055 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-01T02:49:52.055 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-01T02:49:52.055 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:52.058 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-01T02:49:52.148 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-01T02:49:52.149 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-01T02:49:53.438 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-01T02:49:53.438 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 1/34 2024-09-01T02:49:53.438 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 2/34 2024-09-01T02:49:53.438 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 3/34 2024-09-01T02:49:53.438 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 4/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 5/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_6 6/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:19.3.0-4640.g50288870.el9.x86_64 7/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4640.g50288870.el9. 8/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:19.3.0-4640.g50288870.el9.x8 9/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 10/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 11/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 12/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 13/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-01T02:49:53.439 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-01T02:49:53.440 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-01T02:49:53.441 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-01T02:49:53.441 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:19.3.0-4640.g50288870.el9.x86_64 31/34 2024-09-01T02:49:53.441 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-01T02:49:53.441 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:19.3.0-4640.g50288870.el9.x86_64 33/34 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:49:54.193 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.194 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-01T02:49:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-01T02:49:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-20.el9.x86_64 2024-09-01T02:49:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-01T02:49:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:54.196 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:49:54.449 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-09-01T02:49:55.041 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:28 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:49:55.181 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:49:55.181 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:49:55.181 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:49:55.181 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:49:55.181 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:19.3.0-4640.g50288870.el9 ceph 50 M 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:19.3.0-4640.g50288870.el9 ceph 161 k 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-09-01T02:49:55.182 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:49:55.183 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 51 M 2024-09-01T02:49:55.183 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 216 M 2024-09-01T02:49:55.183 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:49:55.496 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): libcephsqlite-19.3.0-4640.g50288870.el9. 760 kB/s | 161 kB 00:00 2024-09-01T02:49:55.850 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 180 kB/s | 64 kB 00:00 2024-09-01T02:49:55.908 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 486 kB/s | 304 kB 00:00 2024-09-01T02:49:57.211 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): ceph-test-19.3.0-4640.g50288870.el9.x86_ 26 MB/s | 50 MB 00:01 2024-09-01T02:49:57.213 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:49:57.214 INFO:teuthology.orchestra.run.smithi079.stdout:Total 25 MB/s | 51 MB 00:02 2024-09-01T02:49:57.240 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:49:57.261 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:49:57.261 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:49:57.639 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:49:57.639 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:49:58.167 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:49:58.244 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-01T02:49:58.285 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-01T02:49:58.313 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 3/4 2024-09-01T02:50:03.005 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 3/4 2024-09-01T02:50:03.086 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 4/4 2024-09-01T02:50:05.610 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 4/4 2024-09-01T02:50:05.610 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 1/4 2024-09-01T02:50:05.610 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 2/4 2024-09-01T02:50:05.610 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-01T02:50:05.933 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-01T02:50:05.933 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:05.933 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:05.934 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:05.934 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:05.934 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-01T02:50:05.934 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-01T02:50:05.934 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:05.934 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:06.233 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-09-01T02:50:06.798 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:39 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:06.936 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:19.3.0-4640.g50288870.el9 ceph 6.6 k 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:19.3.0-4640.g50288870.el9 ceph 2.4 M 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:19.3.0-4640.g50288870.el9 ceph 1.5 M 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 290 k 2024-09-01T02:50:06.939 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:19.3.0-4640.g50288870.el9 ceph 4.8 M 2024-09-01T02:50:06.940 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:19.3.0-4640.g50288870.el9 ceph 17 M 2024-09-01T02:50:06.940 INFO:teuthology.orchestra.run.smithi079.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-01T02:50:06.940 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-01T02:50:06.940 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-01T02:50:06.940 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-01T02:50:06.941 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-01T02:50:06.942 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-01T02:50:06.943 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-01T02:50:06.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:06.944 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:06.944 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:06.944 INFO:teuthology.orchestra.run.smithi079.stdout:Install 37 Packages 2024-09-01T02:50:06.944 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:06.945 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 31 M 2024-09-01T02:50:06.946 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 109 M 2024-09-01T02:50:06.946 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:07.908 INFO:teuthology.orchestra.run.smithi079.stdout:(1/37): ceph-19.3.0-4640.g50288870.el9.x86_64.r 50 kB/s | 6.6 kB 00:00 2024-09-01T02:50:08.083 INFO:teuthology.orchestra.run.smithi079.stdout:(2/37): ceph-mgr-19.3.0-4640.g50288870.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-01T02:50:08.125 INFO:teuthology.orchestra.run.smithi079.stdout:(3/37): ceph-mds-19.3.0-4640.g50288870.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-01T02:50:08.192 INFO:teuthology.orchestra.run.smithi079.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4640.g5028 4.3 MB/s | 290 kB 00:00 2024-09-01T02:50:08.317 INFO:teuthology.orchestra.run.smithi079.stdout:(5/37): ceph-mon-19.3.0-4640.g50288870.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-01T02:50:08.751 INFO:teuthology.orchestra.run.smithi079.stdout:(6/37): ceph-osd-19.3.0-4640.g50288870.el9.x86_ 26 MB/s | 17 MB 00:00 2024-09-01T02:50:09.318 INFO:teuthology.orchestra.run.smithi079.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 225 kB/s | 253 kB 00:01 2024-09-01T02:50:09.477 INFO:teuthology.orchestra.run.smithi079.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 147 kB/s | 106 kB 00:00 2024-09-01T02:50:10.060 INFO:teuthology.orchestra.run.smithi079.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 182 kB/s | 135 kB 00:00 2024-09-01T02:50:10.169 INFO:teuthology.orchestra.run.smithi079.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 180 kB/s | 125 kB 00:00 2024-09-01T02:50:10.236 INFO:teuthology.orchestra.run.smithi079.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 654 kB/s | 1.2 MB 00:01 2024-09-01T02:50:10.311 INFO:teuthology.orchestra.run.smithi079.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-01T02:50:10.361 INFO:teuthology.orchestra.run.smithi079.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 821 kB/s | 35 kB 00:00 2024-09-01T02:50:10.387 INFO:teuthology.orchestra.run.smithi079.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-01T02:50:10.437 INFO:teuthology.orchestra.run.smithi079.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-01T02:50:10.462 INFO:teuthology.orchestra.run.smithi079.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-09-01T02:50:10.488 INFO:teuthology.orchestra.run.smithi079.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 842 kB/s | 42 kB 00:00 2024-09-01T02:50:10.530 INFO:teuthology.orchestra.run.smithi079.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-01T02:50:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-01T02:50:10.589 INFO:teuthology.orchestra.run.smithi079.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-09-01T02:50:10.623 INFO:teuthology.orchestra.run.smithi079.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-01T02:50:10.648 INFO:teuthology.orchestra.run.smithi079.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-01T02:50:10.673 INFO:teuthology.orchestra.run.smithi079.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 705 kB/s | 18 kB 00:00 2024-09-01T02:50:10.698 INFO:teuthology.orchestra.run.smithi079.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 928 kB/s | 23 kB 00:00 2024-09-01T02:50:10.724 INFO:teuthology.orchestra.run.smithi079.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 776 kB/s | 19 kB 00:00 2024-09-01T02:50:10.749 INFO:teuthology.orchestra.run.smithi079.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 789 kB/s | 20 kB 00:00 2024-09-01T02:50:10.775 INFO:teuthology.orchestra.run.smithi079.stdout:(27/37): lua-devel-5.4.4-4.el9.x86_64.rpm 71 kB/s | 22 kB 00:00 2024-09-01T02:50:10.800 INFO:teuthology.orchestra.run.smithi079.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-01T02:50:10.842 INFO:teuthology.orchestra.run.smithi079.stdout:(29/37): python3-pecan-1.4.2-3.el9.noarch.rpm 7.9 MB/s | 272 kB 00:00 2024-09-01T02:50:10.867 INFO:teuthology.orchestra.run.smithi079.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 711 kB/s | 16 kB 00:00 2024-09-01T02:50:10.893 INFO:teuthology.orchestra.run.smithi079.stdout:(31/37): python3-more-itertools-8.12.0-2.el9.no 670 kB/s | 79 kB 00:00 2024-09-01T02:50:10.918 INFO:teuthology.orchestra.run.smithi079.stdout:(32/37): python3-urllib3-1.26.5-6.el9.noarch.rp 250 kB/s | 215 kB 00:00 2024-09-01T02:50:10.943 INFO:teuthology.orchestra.run.smithi079.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 729 kB/s | 36 kB 00:00 2024-09-01T02:50:10.977 INFO:teuthology.orchestra.run.smithi079.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 822 kB/s | 90 kB 00:00 2024-09-01T02:50:11.002 INFO:teuthology.orchestra.run.smithi079.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 866 kB/s | 20 kB 00:00 2024-09-01T02:50:11.027 INFO:teuthology.orchestra.run.smithi079.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-01T02:50:11.052 INFO:teuthology.orchestra.run.smithi079.stdout:(37/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-09-01T02:50:11.057 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:11.057 INFO:teuthology.orchestra.run.smithi079.stdout:Total 7.5 MB/s | 31 MB 00:04 2024-09-01T02:50:11.213 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:11.253 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:11.253 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:11.555 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:11.557 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:12.962 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:13.031 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-01T02:50:13.064 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-01T02:50:13.092 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-01T02:50:13.163 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-01T02:50:13.200 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-01T02:50:13.232 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-01T02:50:13.325 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-01T02:50:13.395 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-01T02:50:13.435 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-01T02:50:13.470 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-01T02:50:13.500 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-01T02:50:13.535 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-01T02:50:13.568 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-01T02:50:13.618 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-01T02:50:13.652 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-01T02:50:13.687 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-01T02:50:13.745 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-01T02:50:13.784 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-01T02:50:13.858 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-01T02:50:14.005 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-01T02:50:14.084 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-01T02:50:14.139 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-01T02:50:14.177 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-01T02:50:14.243 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-01T02:50:14.316 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-01T02:50:14.511 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-01T02:50:14.557 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-01T02:50:14.636 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-01T02:50:14.724 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-01T02:50:14.765 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-01T02:50:14.894 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-01T02:50:15.063 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4640.g50288870.el9. 32/37 2024-09-01T02:50:15.101 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 33/37 2024-09-01T02:50:15.127 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 33/37 2024-09-01T02:50:15.127 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:50:15.127 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-01T02:50:15.127 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-01T02:50:15.127 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-01T02:50:15.128 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:16.593 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 34/37 2024-09-01T02:50:16.620 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 34/37 2024-09-01T02:50:16.620 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:50:16.620 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-01T02:50:16.621 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-01T02:50:16.621 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-01T02:50:16.621 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:17.078 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 35/37 2024-09-01T02:50:17.102 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 35/37 2024-09-01T02:50:17.102 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:50:17.102 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-01T02:50:17.102 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-01T02:50:17.102 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-01T02:50:17.102 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:17.328 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 36/37 2024-09-01T02:50:17.352 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 36/37 2024-09-01T02:50:17.352 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:50:17.352 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-01T02:50:17.353 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-01T02:50:17.353 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-01T02:50:17.353 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:17.424 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:19.3.0-4640.g50288870.el9.x86_64 37/37 2024-09-01T02:50:19.144 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:19.3.0-4640.g50288870.el9.x86_64 37/37 2024-09-01T02:50:19.144 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:19.3.0-4640.g50288870.el9.x86_64 1/37 2024-09-01T02:50:19.144 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 2/37 2024-09-01T02:50:19.144 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 3/37 2024-09-01T02:50:19.144 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 4/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 5/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4640.g50288870.el9. 6/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-01T02:50:19.145 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-01T02:50:19.146 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-01T02:50:19.147 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-01T02:50:20.108 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-01T02:50:20.108 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:20.108 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: lua-5.4.4-4.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-01T02:50:20.109 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-01T02:50:20.110 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-01T02:50:20.111 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:20.112 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:20.510 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-09-01T02:50:21.069 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:54 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:21.159 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:50:21.201 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:21.202 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:50:21.202 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:21.279 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-09-01T02:50:21.828 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:54 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:21.959 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 783 k 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:21.960 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:21.961 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:50:21.961 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:21.961 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 783 k 2024-09-01T02:50:21.961 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 789 k 2024-09-01T02:50:21.961 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:22.235 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-19.3.0-4640.g50288870.el9.noarch.rpm 2.8 MB/s | 783 kB 00:00 2024-09-01T02:50:22.236 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:22.236 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.8 MB/s | 783 kB 00:00 2024-09-01T02:50:22.236 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:22.241 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:22.241 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:22.247 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:22.248 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:22.319 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:22.797 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:50:22.889 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:50:23.254 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:50:23.504 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:50:23.504 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:23.504 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:23.504 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:23.504 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:23.504 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:23.714 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-09-01T02:50:24.270 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:57 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:24.404 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4640.g50288870.el9 ceph 145 k 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:50:24.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:24.406 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 145 k 2024-09-01T02:50:24.406 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 432 k 2024-09-01T02:50:24.406 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:24.593 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-19.3.0-4640.g502888 774 kB/s | 145 kB 00:00 2024-09-01T02:50:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:Total 767 kB/s | 145 kB 00:00 2024-09-01T02:50:24.594 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:24.602 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:24.602 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:24.658 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:24.659 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:24.804 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:24.888 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4640.g50288870. 1/1 2024-09-01T02:50:24.914 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4640.g50288870. 1/1 2024-09-01T02:50:24.914 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:50:24.914 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-01T02:50:24.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:25.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4640.g50288870. 1/1 2024-09-01T02:50:25.848 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:25.848 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:25.848 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:50:25.848 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:25.849 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:26.077 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-09-01T02:50:26.636 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:59 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:26.729 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:50:26.772 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:26.773 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:50:26.774 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:26.850 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-09-01T02:50:27.398 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:00 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:27.529 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:27.530 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:27.530 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:27.530 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:27.530 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:27.530 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 14 M 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 34 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 16 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-01T02:50:27.531 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 19 M 2024-09-01T02:50:27.532 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 158 M 2024-09-01T02:50:27.533 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:28.343 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4640.g50 96 kB/s | 34 kB 00:00 2024-09-01T02:50:28.369 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4640.g502 43 kB/s | 16 kB 00:00 2024-09-01T02:50:28.580 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-01T02:50:28.678 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-09-01T02:50:28.728 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-01T02:50:28.770 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-09-01T02:50:28.788 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 330 kB/s | 19 kB 00:00 2024-09-01T02:50:28.821 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.5 MB/s | 144 kB 00:00 2024-09-01T02:50:28.846 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-01T02:50:28.871 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-01T02:50:28.930 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 13 MB/s | 2.0 MB 00:00 2024-09-01T02:50:29.722 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4640.g502888 8.1 MB/s | 14 MB 00:01 2024-09-01T02:50:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:Total 8.8 MB/s | 19 MB 00:02 2024-09-01T02:50:29.779 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:29.799 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:29.799 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:30.020 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:30.021 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:30.624 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:30.723 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-01T02:50:30.766 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-01T02:50:30.813 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-01T02:50:30.852 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-01T02:50:30.927 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-01T02:50:31.062 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-01T02:50:31.156 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-01T02:50:31.207 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-01T02:50:31.236 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-01T02:50:31.269 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4640.g50288870.el9 10/12 2024-09-01T02:50:34.471 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4640.g50288870.el 11/12 2024-09-01T02:50:34.532 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noa 12/12 2024-09-01T02:50:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noa 12/12 2024-09-01T02:50:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4640.g50288870.el 1/12 2024-09-01T02:50:36.517 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noa 2/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4640.g50288870.el9 3/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-01T02:50:36.518 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-01T02:50:36.825 INFO:teuthology.orchestra.run.smithi079.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:36.826 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:37.095 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-01T02:50:37.643 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:10 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:37.782 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 7.4 M 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-01T02:50:37.783 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-09-01T02:50:37.784 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-09-01T02:50:37.785 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:37.785 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 38 M 2024-09-01T02:50:37.785 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 212 M 2024-09-01T02:50:37.785 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:38.109 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-01T02:50:38.142 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-01T02:50:38.226 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-01T02:50:38.276 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-01T02:50:38.335 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-01T02:50:38.485 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 13 MB/s | 7.4 MB 00:00 2024-09-01T02:50:38.669 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.7 MB/s | 3.0 MB 00:00 2024-09-01T02:50:38.802 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 772 kB/s | 245 kB 00:00 2024-09-01T02:50:38.894 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-01T02:50:38.970 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.8 MB/s | 4.9 MB 00:00 2024-09-01T02:50:39.170 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-09-01T02:50:39.978 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-09-01T02:50:39.979 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:39.979 INFO:teuthology.orchestra.run.smithi079.stdout:Total 17 MB/s | 38 MB 00:02 2024-09-01T02:50:40.332 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:40.394 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:40.394 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:40.769 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:40.769 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:41.527 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:41.591 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-01T02:50:41.635 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-01T02:50:41.741 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-01T02:50:41.778 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-01T02:50:41.975 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-01T02:50:42.002 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-01T02:50:42.079 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-01T02:50:42.719 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-01T02:50:42.841 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-01T02:50:44.056 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-01T02:50:45.089 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-01T02:50:45.113 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288 12/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288 12/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288 1/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-01T02:50:47.697 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-01T02:50:47.698 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-01T02:50:47.698 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-01T02:50:47.698 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-01T02:50:47.698 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-01T02:50:47.698 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-01T02:50:48.136 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:48.137 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:48.533 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-09-01T02:50:49.113 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:22 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:49.247 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:49.248 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:49.248 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:49.248 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 49 k 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-01T02:50:49.249 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-01T02:50:49.250 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-01T02:50:49.250 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-01T02:50:49.250 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-01T02:50:49.250 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-01T02:50:49.250 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-01T02:50:49.250 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-01T02:50:49.251 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-01T02:50:49.251 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-01T02:50:49.251 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:49.251 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:49.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:49.251 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-09-01T02:50:49.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:49.253 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.2 M 2024-09-01T02:50:49.253 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-09-01T02:50:49.253 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:49.875 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 191 kB/s | 19 kB 00:00 2024-09-01T02:50:49.900 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 204 kB/s | 26 kB 00:00 2024-09-01T02:50:49.934 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): ceph-mgr-rook-19.3.0-4640.g50288870.el9 310 kB/s | 49 kB 00:00 2024-09-01T02:50:50.018 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-01T02:50:50.043 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-01T02:50:50.068 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-01T02:50:50.110 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-01T02:50:50.135 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 235 kB/s | 14 kB 00:00 2024-09-01T02:50:50.160 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-01T02:50:50.203 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-01T02:50:50.236 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.8 MB/s | 90 kB 00:00 2024-09-01T02:50:50.262 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-09-01T02:50:50.303 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-01T02:50:50.307 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:50.307 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-09-01T02:50:50.373 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:50.389 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:50.389 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:50.477 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:50.477 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:50.976 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:51.141 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-01T02:50:51.263 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-01T02:50:51.327 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-01T02:50:51.365 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-01T02:50:51.406 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-01T02:50:51.507 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-01T02:50:51.609 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-01T02:50:51.658 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-01T02:50:52.370 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-01T02:50:52.415 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-01T02:50:52.448 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-01T02:50:52.496 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-01T02:50:52.519 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 13/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 13/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 1/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-01T02:50:53.262 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-01T02:50:53.263 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-01T02:50:53.263 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-01T02:50:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-01T02:50:53.572 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:53.572 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-01T02:50:53.573 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-01T02:50:53.574 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:53.574 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:53.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-09-01T02:50:54.367 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:27 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:54.499 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:54.500 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:54.500 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:54.500 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:54.500 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:54.500 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 154 k 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-09-01T02:50:54.501 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:54.502 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.0 M 2024-09-01T02:50:54.502 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-09-01T02:50:54.502 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:55.234 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-01T02:50:55.293 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-01T02:50:55.318 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-01T02:50:55.343 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-01T02:50:55.393 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): ceph-mgr-cephadm-19.3.0-4640.g50288870.e 414 kB/s | 154 kB 00:00 2024-09-01T02:50:55.652 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.5 MB/s | 6.0 MB 00:00 2024-09-01T02:50:55.654 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:55.654 INFO:teuthology.orchestra.run.smithi079.stdout:Total 6.1 MB/s | 7.0 MB 00:01 2024-09-01T02:50:55.727 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:55.738 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:55.739 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:55.839 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:55.839 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:56.154 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:56.326 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-01T02:50:56.396 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-01T02:50:56.927 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-01T02:50:57.016 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-01T02:50:57.083 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-01T02:50:57.117 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 6/6 2024-09-01T02:50:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 6/6 2024-09-01T02:50:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 1/6 2024-09-01T02:50:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-01T02:50:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-01T02:50:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-01T02:50:57.891 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:58.157 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:50:58.392 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-09-01T02:50:58.941 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:31 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:50:59.072 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:19.3.0-4640.g50288870.el9 ceph 900 k 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-09-01T02:50:59.073 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:50:59.074 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 979 k 2024-09-01T02:50:59.074 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.9 M 2024-09-01T02:50:59.074 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:50:59.297 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 651 kB/s | 80 kB 00:00 2024-09-01T02:50:59.464 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-19.3.0-4640.g50288870.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-01T02:50:59.465 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:50:59.465 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.4 MB/s | 979 kB 00:00 2024-09-01T02:50:59.477 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:50:59.499 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:50:59.499 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:50:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:50:59.580 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:50:59.783 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:50:59.918 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-01T02:50:59.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 2/2 2024-09-01T02:51:00.658 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 2/2 2024-09-01T02:51:00.658 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/2 2024-09-01T02:51:00.923 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-01T02:51:00.923 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:00.923 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:00.924 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:19.3.0-4640.g50288870.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-01T02:51:00.924 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:00.924 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:01.184 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-volume 2024-09-01T02:51:01.748 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:34 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:01.881 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume noarch 2:19.3.0-4640.g50288870.el9 ceph-noarch 285 k 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:51:01.882 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:01.883 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:51:01.883 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:01.883 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 285 k 2024-09-01T02:51:01.883 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 1.3 M 2024-09-01T02:51:01.883 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:51:02.097 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-volume-19.3.0-4640.g50288870.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-01T02:51:02.098 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:51:02.098 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-01T02:51:02.098 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:51:02.107 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:51:02.107 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:51:02.129 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:51:02.130 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:51:02.351 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:51:02.458 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:51:02.474 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:51:02.474 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:51:02.475 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-01T02:51:02.475 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:03.473 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 1/1 2024-09-01T02:51:03.473 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:03.473 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:03.473 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-volume-2:19.3.0-4640.g50288870.el9.noarch 2024-09-01T02:51:03.473 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:03.473 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:03.778 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-09-01T02:51:04.345 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:37 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:04.479 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:04.480 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:04.480 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-01T02:51:04.480 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:04.480 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:51:04.480 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:19.3.0-4640.g50288870.el9 ceph 126 k 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 126 k 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 456 k 2024-09-01T02:51:04.481 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:51:04.672 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-19.3.0-4640.g50288870.el9.x86_64 663 kB/s | 126 kB 00:00 2024-09-01T02:51:04.672 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:51:04.673 INFO:teuthology.orchestra.run.smithi079.stdout:Total 659 kB/s | 126 kB 00:00 2024-09-01T02:51:04.673 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:51:04.678 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:51:04.678 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:51:04.741 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:51:04.741 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:51:04.890 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:51:04.994 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:05.446 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:05.683 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:05.684 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:05.684 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:05.684 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:51:05.684 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:05.684 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:05.908 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-09-01T02:51:06.477 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:39 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:06.569 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:06.614 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:06.615 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:06.615 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:06.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-09-01T02:51:07.248 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:40 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:07.385 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:19.3.0-4640.g50288870.el9 ceph 32 k 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:51:07.386 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:07.387 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 32 k 2024-09-01T02:51:07.387 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 142 k 2024-09-01T02:51:07.387 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:51:07.719 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-19.3.0-4640.g50288870.el9.x86_6 96 kB/s | 32 kB 00:00 2024-09-01T02:51:07.720 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:51:07.720 INFO:teuthology.orchestra.run.smithi079.stdout:Total 95 kB/s | 32 kB 00:00 2024-09-01T02:51:07.720 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:51:07.724 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:51:07.724 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:51:07.729 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:51:07.729 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:51:07.814 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:51:07.915 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:08.292 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:08.517 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:08.517 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:08.517 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:08.517 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:51:08.517 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:08.517 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:08.716 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-09-01T02:51:09.272 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:42 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:09.361 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:09.405 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:09.406 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:09.406 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:09.489 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-09-01T02:51:10.041 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:43 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:10.132 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:10.174 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:10.176 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:10.176 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:10.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-09-01T02:51:10.808 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:43 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:10.899 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:10.943 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:10.944 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:10.944 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:11.046 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-09-01T02:51:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:44 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:11.692 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:11.735 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:11.736 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:11.736 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:11.815 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-09-01T02:51:12.372 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:45 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:12.460 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:12.503 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:12.504 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:12.505 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:12.587 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-09-01T02:51:13.133 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:46 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:13.221 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:19.3.0-4640.g50288870.el9.x86_64 is already installed. 2024-09-01T02:51:13.264 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:13.265 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-09-01T02:51:13.265 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:13.354 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-09-01T02:51:13.907 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:46 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:14.040 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:19.3.0-4640.g50288870.el9 ceph 85 k 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:14.041 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:51:14.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:14.042 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 85 k 2024-09-01T02:51:14.042 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 231 k 2024-09-01T02:51:14.042 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:51:14.256 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-19.3.0-4640.g50288870.el9.x86_64.rpm 399 kB/s | 85 kB 00:00 2024-09-01T02:51:14.257 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:51:14.257 INFO:teuthology.orchestra.run.smithi079.stdout:Total 396 kB/s | 85 kB 00:00 2024-09-01T02:51:14.257 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:51:14.264 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:51:14.264 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:51:14.327 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:51:14.328 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:51:14.466 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:51:14.560 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:14.986 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:15.230 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:15.230 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:15.230 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:15.230 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:51:15.230 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:15.230 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:15.518 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-09-01T02:51:16.082 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:49 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:16.217 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:16.218 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:16.218 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-09-01T02:51:16.218 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:16.219 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:51:16.219 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:19.3.0-4640.g50288870.el9 ceph 3.2 M 2024-09-01T02:51:16.219 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:16.219 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:51:16.219 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:16.219 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:51:16.220 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:16.220 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.2 M 2024-09-01T02:51:16.220 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 14 M 2024-09-01T02:51:16.220 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:51:16.537 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-19.3.0-4640.g50288870.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-01T02:51:16.539 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:51:16.540 INFO:teuthology.orchestra.run.smithi079.stdout:Total 9.9 MB/s | 3.2 MB 00:00 2024-09-01T02:51:16.540 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:51:16.547 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:51:16.547 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:51:16.630 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:51:16.630 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:51:17.069 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:51:17.138 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:17.163 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-01T02:51:17.163 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-01T02:51:17.163 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-01T02:51:17.163 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-01T02:51:17.163 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:18.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:18.258 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:18.258 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:18.258 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:51:18.258 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:18.258 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:18.543 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-09-01T02:51:19.100 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:52 ago on Sun 01 Sep 2024 02:49:27 AM UTC. 2024-09-01T02:51:19.234 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:19.3.0-4640.g50288870.el9 ceph 171 k 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-01T02:51:19.235 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-09-01T02:51:19.236 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:19.236 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 171 k 2024-09-01T02:51:19.236 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 490 k 2024-09-01T02:51:19.236 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-01T02:51:19.443 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-19.3.0-4640.g50288870.el9.x86_64.rpm 825 kB/s | 171 kB 00:00 2024-09-01T02:51:19.444 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-01T02:51:19.444 INFO:teuthology.orchestra.run.smithi079.stdout:Total 819 kB/s | 171 kB 00:00 2024-09-01T02:51:19.444 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-01T02:51:19.452 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-01T02:51:19.452 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-01T02:51:19.515 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-01T02:51:19.515 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-01T02:51:19.667 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-01T02:51:19.765 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:20.198 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:20.443 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 1/1 2024-09-01T02:51:20.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:20.443 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-01T02:51:20.443 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:19.3.0-4640.g50288870.el9.x86_64 2024-09-01T02:51:20.443 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-01T02:51:20.443 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-01T02:51:20.673 DEBUG:teuthology.parallel:result is None 2024-09-01T02:51:20.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T02:51:20.824 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-01T02:51:20.845 INFO:teuthology.orchestra.run.smithi079.stdout:19.3.0-4640.g50288870.el9 2024-09-01T02:51:20.846 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4640.g50288870.el9 2024-09-01T02:51:20.846 INFO:teuthology.task.install:The correct ceph version 19.3.0-4640.g50288870 is installed. 2024-09-01T02:51:20.848 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-01T02:51:20.848 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:51:20.848 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-01T02:51:20.911 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-01T02:51:20.911 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:51:20.912 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-09-01T02:51:20.976 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-01T02:51:21.040 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-01T02:51:21.041 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:51:21.041 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-01T02:51:21.104 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-01T02:51:21.169 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-01T02:51:21.169 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:51:21.169 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-09-01T02:51:21.234 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-01T02:51:21.299 INFO:teuthology.run_tasks:Running task workunit... 2024-09-01T02:51:21.314 INFO:tasks.workunit:Pulling workunits from ref 5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T02:51:21.314 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-01T02:51:21.314 INFO:tasks.workunit:timeout=3h 2024-09-01T02:51:21.314 INFO:tasks.workunit:cleanup=True 2024-09-01T02:51:21.315 DEBUG:teuthology.orchestra.run.smithi079:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-01T02:51:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T02:51:21.355 INFO:teuthology.orchestra.run.smithi079.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-01T02:51:21.355 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-01T02:51:21.410 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-01T02:51:21.410 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-01T02:51:21.466 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5028887063e8eb518c95117479aa913eaa12ac8e 2024-09-01T02:51:21.522 INFO:tasks.workunit.client.0.smithi079.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-01T02:52:10.886 INFO:tasks.workunit.client.0.smithi079.stderr:Note: switching to '5028887063e8eb518c95117479aa913eaa12ac8e'. 2024-09-01T02:52:10.886 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr:state without impacting any branches by switching back to a branch. 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr: git switch -c 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr:Or undo this operation with: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.887 INFO:tasks.workunit.client.0.smithi079.stderr: git switch - 2024-09-01T02:52:10.888 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.888 INFO:tasks.workunit.client.0.smithi079.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-01T02:52:10.888 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-09-01T02:52:10.888 INFO:tasks.workunit.client.0.smithi079.stderr:HEAD is now at 5028887063e Merge branch 'wip-bug-67087' of https://github.com/rzarzynski/ceph into wip-bharath2-testing-2024-08-31-2129 2024-09-01T02:52:10.890 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-01T02:52:10.946 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-01T02:52:10.946 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-01T02:52:11.002 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-01T02:52:11.002 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-01T02:52:11.002 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-01T02:52:11.067 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/bad-inc-map 2024-09-01T02:52:11.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-01T02:52:11.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-01T02:52:11.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:57619 2024-09-01T02:52:11.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:57619 2024-09-01T02:52:11.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-01T02:52:11.098 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-01T02:52:11.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none ' 2024-09-01T02:52:11.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:57619 ' 2024-09-01T02:52:11.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-01T02:52:11.106 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-01T02:52:11.106 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/bad-inc-map 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/bad-inc-map 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:52:11.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:52:11.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:52:11.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:52:11.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:52:11.110 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:52:11.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:52:11.111 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:52:11.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:52:11.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:52:11.112 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:52:11.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:52:11.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:52:11.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:52:11.114 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:52:11.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:52:11.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:52:11.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-01T02:52:11.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:52:11.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:11.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:11.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16177 2024-09-01T02:52:11.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:52:11.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:52:11.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/bad-inc-map 2024-09-01T02:52:11.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:52:11.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:11.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:11.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.16177 2024-09-01T02:52:11.120 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:52:11.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/bad-inc-map 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/bad-inc-map/a 2024-09-01T02:52:11.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-01T02:52:11.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:52:11.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:52:11.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:52:11.691 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:52:11.691 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:11.691 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:11.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:52:11.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T02:52:11.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:52:11.876 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:52:11.877 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:52:11.877 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:11.877 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:11.877 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16177/ceph-mon.a.asok 2024-09-01T02:52:11.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:52:11.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16177/ceph-mon.a.asok config get fsid 2024-09-01T02:52:11.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:52:11.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:52:11.968 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.16177/ceph-mon.a.asok 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:52:11.969 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.16177/ceph-mon.a.asok config get mon_host 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/bad-inc-map 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/bad-inc-map/x 2024-09-01T02:52:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:52:12.887 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:52:12.887 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:52:12.887 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:52:12.887 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:52:12.888 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:12.888 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:12.888 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:52:12.888 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:52:12.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:52:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-01T02:52:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/0 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 ' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:52:12.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-01T02:52:12.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:52:12.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:52:12.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:52:12.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:52:12.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:52:12.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:52:12.920 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/0 2024-09-01T02:52:12.921 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:52:12.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=33fcd34b-90ad-42e1-b4e9-f1f1cb7868df 2024-09-01T02:52:12.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 33fcd34b-90ad-42e1-b4e9-f1f1cb7868df' 2024-09-01T02:52:12.921 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 33fcd34b-90ad-42e1-b4e9-f1f1cb7868df 2024-09-01T02:52:12.922 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:52:12.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDc1tNmmdgOOBAAVZgiiUBgyOsY0UeB21Eu1g== 2024-09-01T02:52:12.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDc1tNmmdgOOBAAVZgiiUBgyOsY0UeB21Eu1g=="}' 2024-09-01T02:52:12.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 33fcd34b-90ad-42e1-b4e9-f1f1cb7868df -i td/bad-inc-map/0/new.json 2024-09-01T02:52:13.218 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:52:13.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/0/new.json 2024-09-01T02:52:13.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDc1tNmmdgOOBAAVZgiiUBgyOsY0UeB21Eu1g== --osd-uuid 33fcd34b-90ad-42e1-b4e9-f1f1cb7868df 2024-09-01T02:52:13.258 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:13.256+0000 7f7f0bd51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:13.287 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:13.285+0000 7f7f0bd51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:13.288 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:13.286+0000 7f7f0bd51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:13.288 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:13.287+0000 7f7f0bd51640 -1 bdev(0x5612c0b41c00 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:52:13.288 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:13.287+0000 7f7f0bd51640 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-01T02:52:19.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-01T02:52:19.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:52:19.593 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T02:52:19.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:52:19.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:52:20.119 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T02:52:20.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:52:20.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:52:20.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:52:20.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:52:20.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:52:20.144 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:20.143+0000 7f7efd94f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:20.164 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:20.162+0000 7f7efd94f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:20.165 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:20.163+0000 7f7efd94f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:52:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:52:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:52:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:52:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:52:20.545 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:52:20.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:20.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:52:20.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:20.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:20.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:21.259 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:21.258+0000 7f7efd94f640 -1 Falling back to public interface 2024-09-01T02:52:21.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:21.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:21.964 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:52:21.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:52:21.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:21.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:22.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:23.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:23.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:23.381 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:52:23.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:52:23.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:23.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:23.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:24.317 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:24.316+0000 7f7efd94f640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:52:24.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:24.771 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:52:24.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:24.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:52:24.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:24.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:25.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:26.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:26.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:26.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:52:26.265 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:52:26.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:26.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:26.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:27.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:27.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:27.685 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:52:27.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:52:27.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:27.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:29.216 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:52:29.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:52:29.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:29.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:29.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:30.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:30.655 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:52:30.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:30.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:52:30.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:30.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:31.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:32.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:32.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:32.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:52:32.087 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:52:32.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:32.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:32.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:33.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:33.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:33.508 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:52:33.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:52:33.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:33.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:33.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:34.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:34.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:34.898 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:52:34.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:52:34.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:34.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:35.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:36.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:36.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:36.289 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:52:36.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:52:36.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:36.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:36.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:37.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:37.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:37.931 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:52:37.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:52:37.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:37.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:38.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:39.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:39.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:39.364 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:52:39.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:52:39.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:39.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:40.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:41.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:41.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:41.010 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:52:41.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:52:41.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:41.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:41.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:42.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:42.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:42.646 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:52:42.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:52:42.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:42.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:43.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:44.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:44.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:44.043 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:52:44.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:52:44.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:44.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:44.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:45.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:45.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:45.729 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:52:45.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:52:45.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:45.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:46.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:47.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:47.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:47.707 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T02:52:47.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:52:47.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:47.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:48.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:49.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:49.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:49.619 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-09-01T02:52:49.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T02:52:49.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:49.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:52:50.048 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3356170873,v1:127.0.0.1:6803/3356170873] [v2:127.0.0.1:6804/3356170873,v1:127.0.0.1:6805/3356170873] exists,up 33fcd34b-90ad-42e1-b4e9-f1f1cb7868df 2024-09-01T02:52:50.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/1 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 ' 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:52:50.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:52:50.054 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:52:50.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:52:50.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:52:50.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/1 2024-09-01T02:52:50.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:52:50.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=948ec97d-02ca-49b7-ae32-fde7c71ba2a6 2024-09-01T02:52:50.056 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 948ec97d-02ca-49b7-ae32-fde7c71ba2a6 2024-09-01T02:52:50.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 948ec97d-02ca-49b7-ae32-fde7c71ba2a6' 2024-09-01T02:52:50.057 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:52:50.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAC19Nmpd//AxAArz25zJWpVuRv7UXhydt6hg== 2024-09-01T02:52:50.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAC19Nmpd//AxAArz25zJWpVuRv7UXhydt6hg=="}' 2024-09-01T02:52:50.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 948ec97d-02ca-49b7-ae32-fde7c71ba2a6 -i td/bad-inc-map/1/new.json 2024-09-01T02:52:50.706 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:52:50.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/1/new.json 2024-09-01T02:52:50.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAC19Nmpd//AxAArz25zJWpVuRv7UXhydt6hg== --osd-uuid 948ec97d-02ca-49b7-ae32-fde7c71ba2a6 2024-09-01T02:52:50.745 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:50.743+0000 7efcccb19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:50.748 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:50.745+0000 7efcccb19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:50.749 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:50.746+0000 7efcccb19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:50.749 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:50.746+0000 7efcccb19640 -1 bdev(0x5591c6381c00 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-09-01T02:52:50.749 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:50.747+0000 7efcccb19640 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-01T02:52:56.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-01T02:52:56.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:52:56.403 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T02:52:56.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T02:52:56.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:52:56.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T02:52:56.935 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T02:52:56.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:52:56.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:52:56.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:52:56.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:52:56.959 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:56.958+0000 7fe9b7e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:56.976 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:56.974+0000 7fe9b7e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:56.977 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:56.975+0000 7fe9b7e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:52:57.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T02:52:57.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:52:57.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:52:57.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:52:57.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:52:57.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:57.349 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:52:57.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:52:57.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:57.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:52:57.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:52:58.325 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:52:58.324+0000 7fe9b7e0e640 -1 Falling back to public interface 2024-09-01T02:52:58.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:52:58.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:52:58.767 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:52:58.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:52:58.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:52:58.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:52:59.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:00.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:00.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:00.175 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:53:00.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:53:00.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:00.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:00.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:01.014 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:01.013+0000 7fe9b7e0e640 -1 osd.1 0 log_to_monitors true 2024-09-01T02:53:01.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:01.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:01.599 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:53:01.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:53:01.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:01.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:02.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:03.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:03.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:03.087 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:53:03.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:53:03.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:03.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:03.608 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:04.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:04.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:04.610 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:53:04.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:53:04.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:04.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:05.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:06.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:06.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:06.091 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:53:06.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:53:06.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:06.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:06.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:07.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:07.496 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:53:07.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:07.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:53:07.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:07.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:07.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:08.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:08.963 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:53:08.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:08.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:53:08.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:08.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:09.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:10.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:10.375 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:53:10.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:10.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:53:10.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:10.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:10.866 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:11.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:11.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:11.868 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:53:11.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:53:11.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:11.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:12.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:13.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:13.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:13.258 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:53:13.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:53:13.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:13.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:13.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:14.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:14.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:14.899 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:53:14.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:53:14.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:14.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:15.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:16.677 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:53:16.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:53:16.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:16.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:17.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:18.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:18.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:18.487 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:53:18.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:53:18.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:18.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:19.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:20.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:20.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:20.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:53:20.037 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:53:20.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:20.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:20.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:21.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:21.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:21.784 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:53:21.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:53:21.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:21.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:22.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:23.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:23.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:23.436 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:53:23.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:53:23.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:23.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:24.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:25.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:25.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:25.457 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T02:53:25.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:53:25.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:25.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:53:25.877 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2045843485,v1:127.0.0.1:6811/2045843485] [v2:127.0.0.1:6812/2045843485,v1:127.0.0.1:6813/2045843485] exists,up 948ec97d-02ca-49b7-ae32-fde7c71ba2a6 2024-09-01T02:53:25.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:53:25.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:53:25.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:53:25.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-01T02:53:25.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/2 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 ' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:53:25.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-01T02:53:25.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/2 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:53:25.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03af3db8-62a4-4546-b98c-7ef0526261d6 2024-09-01T02:53:25.885 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 03af3db8-62a4-4546-b98c-7ef0526261d6 2024-09-01T02:53:25.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 03af3db8-62a4-4546-b98c-7ef0526261d6' 2024-09-01T02:53:25.885 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:53:25.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAl19Nm7nhsNRAAVCdg4gM2InglCvcdMZ1sbQ== 2024-09-01T02:53:25.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAl19Nm7nhsNRAAVCdg4gM2InglCvcdMZ1sbQ=="}' 2024-09-01T02:53:25.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03af3db8-62a4-4546-b98c-7ef0526261d6 -i td/bad-inc-map/2/new.json 2024-09-01T02:53:26.427 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:53:26.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/2/new.json 2024-09-01T02:53:26.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAl19Nm7nhsNRAAVCdg4gM2InglCvcdMZ1sbQ== --osd-uuid 03af3db8-62a4-4546-b98c-7ef0526261d6 2024-09-01T02:53:26.466 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:26.464+0000 7f25eef43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:53:26.501 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:26.500+0000 7f25eef43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:53:26.502 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:26.501+0000 7f25eef43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:53:26.503 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:26.501+0000 7f25eef43640 -1 bdev(0x555e1433dc00 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-09-01T02:53:26.503 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:26.501+0000 7f25eef43640 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-01T02:53:32.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-01T02:53:32.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:53:32.273 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T02:53:32.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T02:53:32.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:53:32.814 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T02:53:32.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T02:53:32.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=74592d45-0c84-4544-88fd-84ba47e14ef5 --auth-supported=none --mon-host=127.0.0.1:57619 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.16177/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:53:32.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:53:32.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:53:32.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:53:32.839 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:32.838+0000 7ff020664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:53:32.847 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:32.845+0000 7ff020664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:53:32.848 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:32.846+0000 7ff020664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:53:33.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T02:53:33.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:53:33.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T02:53:33.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:53:33.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:53:33.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:33.231 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:53:33.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:53:33.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:33.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:33.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:34.204 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:34.203+0000 7ff020664640 -1 Falling back to public interface 2024-09-01T02:53:34.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:34.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:34.656 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:53:34.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:53:34.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:34.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:35.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:36.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:36.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:36.064 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:53:36.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:53:36.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:36.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:36.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:36.800 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:53:36.798+0000 7ff020664640 -1 osd.2 0 log_to_monitors true 2024-09-01T02:53:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:37.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:37.457 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:53:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:53:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:37.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:37.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:38.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:38.927 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:53:38.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:38.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:53:38.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:38.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:39.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:40.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:40.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:40.378 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:53:40.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:53:40.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:40.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:40.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:41.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:41.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:41.918 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:53:41.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:53:41.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:41.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:42.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:43.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:43.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:43.334 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:53:43.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:53:43.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:43.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:43.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:44.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:44.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:44.837 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:53:44.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:53:44.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:44.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:45.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:46.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:46.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:46.287 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:53:46.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:53:46.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:46.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:46.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:47.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:47.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:47.706 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:53:47.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:53:47.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:47.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:48.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:49.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:49.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:49.124 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:53:49.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:53:49.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:49.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:49.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:50.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:50.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:50.736 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:53:50.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:53:50.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:50.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:51.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:52.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:52.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:52.632 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:53:52.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:53:52.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:52.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:53.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:54.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:54.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:54.079 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:53:54.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:53:54.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:54.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:54.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:55.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:55.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:55.800 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:53:55.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:53:55.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:55.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:56.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:57.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:57.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:57.537 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:53:57.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:53:57.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:57.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:53:58.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:53:59.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:53:59.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:53:59.364 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:53:59.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:53:59.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:53:59.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:54:00.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:01.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:01.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:01.198 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T02:54:01.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:54:01.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:01.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:54:01.586 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/848373409,v1:127.0.0.1:6819/848373409] [v2:127.0.0.1:6820/848373409,v1:127.0.0.1:6821/848373409] exists,up 03af3db8-62a4-4546-b98c-7ef0526261d6 2024-09-01T02:54:01.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:54:01.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:54:01.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:54:01.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-01T02:54:02.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-01T02:54:02.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 8 2024-09-01T02:54:02.663 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-09-01T02:54:02.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:54:03.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-01T02:54:05.035 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 1 2024-09-01T02:54:05.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-01T02:54:06.234 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-09-01T02:54:06.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-01T02:54:11.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-01T02:54:11.248 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:54:11.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:54:11.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:11.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:11.636 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3356170873,v1:127.0.0.1:6803/3356170873] [v2:127.0.0.1:6804/3356170873,v1:127.0.0.1:6805/3356170873] exists,up 33fcd34b-90ad-42e1-b4e9-f1f1cb7868df 2024-09-01T02:54:11.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:54:11.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-01T02:54:11.638 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:54:11.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:54:11.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:11.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:54:12.011 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2045843485,v1:127.0.0.1:6811/2045843485] [v2:127.0.0.1:6812/2045843485,v1:127.0.0.1:6813/2045843485] exists,up 948ec97d-02ca-49b7-ae32-fde7c71ba2a6 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:54:12.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-01T02:54:12.014 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:54:12.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:54:12.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:12.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:54:12.425 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/848373409,v1:127.0.0.1:6819/848373409] [v2:127.0.0.1:6820/848373409,v1:127.0.0.1:6821/848373409] exists,up 03af3db8-62a4-4546-b98c-7ef0526261d6 2024-09-01T02:54:12.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:54:12.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:54:12.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:54:12.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-01T02:54:12.428 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:02.560+0000 7ff0115f0640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.428 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:03.164+0000 7ff0115f0640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.428 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:03.591+0000 7ff0115f0640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:04.497+0000 7ff00f5ec640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:04.891+0000 7ff00f5ec640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:04.984+0000 7ff00f5ec640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:05.085+0000 7ff0115f0640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:05.887+0000 7ff00f5ec640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:06.274+0000 7ff0115f0640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:06.714+0000 7ff0115f0640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:06.715+0000 7ff0115f0640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-01T02:54:12.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:02.560+0000 7ff0115f0640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:03.165+0000 7ff0115f0640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:03.591+0000 7ff0115f0640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:04.497+0000 7ff00f5ec640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:04.891+0000 7ff00f5ec640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:04.984+0000 7ff00f5ec640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:05.085+0000 7ff0115f0640 10 osd.2 21 handle_osd_map bailing because last < start (22<23) 2024-09-01T02:54:12.430 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:05.888+0000 7ff00f5ec640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-01T02:54:12.431 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:06.275+0000 7ff0115f0640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-01T02:54:12.431 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:06.714+0000 7ff0115f0640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-01T02:54:12.431 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T02:54:06.716+0000 7ff0115f0640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-01T02:54:12.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-01T02:54:12.431 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-09-01T02:54:12.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-01T02:54:12.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=foo 2024-09-01T02:54:12.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-01T02:54:12.904 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' does not exist 2024-09-01T02:54:12.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-01T02:54:12.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:54:12.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:54:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:54:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:54:12.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:54:18.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:54:18.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:54:18.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:54:18.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:54:18.236 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:54:18.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:54:18.238 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:54:18.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:54:18.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:54:18.239 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:54:18.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:54:18.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:54:18.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:54:18.241 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:54:18.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:54:18.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:54:18.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-01T02:54:18.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:54:18.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:18.258 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:54:18.258 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16177 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/bad-inc-map 0 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:54:18.260 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:54:18.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:54:18.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:54:18.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:54:18.262 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:54:18.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:54:18.263 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:54:18.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:54:18.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:54:18.264 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:54:18.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:54:18.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:54:18.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:54:18.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:54:18.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:54:18.268 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T02:54:18.268 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-01T02:54:18.269 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:54:18.269 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:18.269 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.16177 2024-09-01T02:54:18.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.16177 2024-09-01T02:54:18.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:54:18.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:54:18.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T02:54:18.271 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T02:54:18.272 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T02:54:18.343 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-01T02:54:18.343 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-01T02:54:18.407 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T02:54:18.413 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/divergent-priors 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-01T02:54:18.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-01T02:54:18.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-01T02:54:18.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-01T02:54:18.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-01T02:54:18.415 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-01T02:54:18.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-01T02:54:18.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-01T02:54:18.416 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-01T02:54:18.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-01T02:54:18.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-01T02:54:18.416 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-01T02:54:18.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-01T02:54:18.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-01T02:54:18.417 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-01T02:54:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-01T02:54:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-01T02:54:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-01T02:54:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-01T02:54:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-01T02:54:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-01T02:54:18.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-01T02:54:18.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-01T02:54:18.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-01T02:54:18.419 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none ' 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-01T02:54:18.420 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T02:54:18.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-01T02:54:18.422 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_divergent_2 2024-09-01T02:54:18.422 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_divergent_3 2024-09-01T02:54:18.422 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_divergent_ec' 2024-09-01T02:54:18.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:54:18.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:54:18.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:54:18.424 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:54:18.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:54:18.425 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:54:18.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:54:18.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:54:18.426 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:54:18.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:54:18.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:54:18.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:54:18.427 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:54:18.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:54:18.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:54:18.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T02:54:18.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:54:18.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:18.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:18.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T02:54:18.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:54:18.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:54:18.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-01T02:54:18.432 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:54:18.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:18.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:18.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.22149 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-01T02:54:18.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-01T02:54:18.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-01T02:54:18.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-01T02:54:18.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-01T02:54:18.436 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-01T02:54:18.436 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:54:18.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:54:18.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:54:18.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-01T02:54:18.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-01T02:54:18.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:54:18.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:54:18.984 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:54:18.984 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:54:18.984 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:18.984 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:18.984 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:54:18.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T02:54:19.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:54:19.185 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:54:19.186 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:54:19.187 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:19.187 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:19.187 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T02:54:19.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:54:19.187 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get fsid 2024-09-01T02:54:19.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:54:19.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:54:19.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:54:19.267 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get mon_host 2024-09-01T02:54:19.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-01T02:54:19.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-01T02:54:19.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:54:19.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:54:19.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:54:19.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-01T02:54:19.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:54:20.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:54:20.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:54:20.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:54:20.205 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:54:20.205 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:20.205 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:20.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:54:20.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:54:20.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:54:20.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:54:20.245 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:54:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=195d3106-9752-4c12-9ff0-98623a3e39c1 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 195d3106-9752-4c12-9ff0-98623a3e39c1' 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 195d3106-9752-4c12-9ff0-98623a3e39c1 2024-09-01T02:54:20.247 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:54:20.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBc19NmRyG9DxAApsfXqtOu1fUdSpApEDXfiQ== 2024-09-01T02:54:20.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBc19NmRyG9DxAApsfXqtOu1fUdSpApEDXfiQ=="}' 2024-09-01T02:54:20.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 195d3106-9752-4c12-9ff0-98623a3e39c1 -i td/divergent-priors/0/new.json 2024-09-01T02:54:20.533 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:54:20.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-01T02:54:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBc19NmRyG9DxAApsfXqtOu1fUdSpApEDXfiQ== --osd-uuid 195d3106-9752-4c12-9ff0-98623a3e39c1 2024-09-01T02:54:20.581 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:20.580+0000 7efc7cc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:20.627 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:20.626+0000 7efc7cc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:20.628 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:20.627+0000 7efc7cc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:20.629 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:20.627+0000 7efc7cc0d640 -1 bdev(0x563ebf029c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:54:20.629 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:20.627+0000 7efc7cc0d640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-01T02:54:26.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-01T02:54:26.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:54:26.439 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T02:54:26.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:54:26.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:54:26.974 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T02:54:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:54:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:54:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:54:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:54:26.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:54:27.009 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:27.008+0000 7ffa4c9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:27.026 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:27.024+0000 7ffa4c9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:27.027 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:27.025+0000 7ffa4c9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:27.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:54:27.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:54:27.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:54:27.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:54:27.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:54:27.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:27.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:54:27.432 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:54:27.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:27.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:27.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:28.632 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:28.631+0000 7ffa4c9b0640 -1 Falling back to public interface 2024-09-01T02:54:28.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:28.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:28.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:54:28.858 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:54:28.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:28.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:30.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:30.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:30.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:54:30.292 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:54:30.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:30.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:30.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:31.329 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:31.327+0000 7ffa4c9b0640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:54:31.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:31.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:31.696 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:54:31.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:54:31.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:31.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:32.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:33.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:33.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:33.105 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:54:33.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:54:33.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:33.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:33.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:34.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:34.528 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:54:34.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:34.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:54:34.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:34.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:34.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:35.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:35.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:35.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:54:35.933 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:54:35.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:35.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:36.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:37.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:37.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:37.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:54:37.498 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:54:37.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:37.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:37.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:38.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:38.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:38.917 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:54:38.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:54:38.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:38.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:39.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:40.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:40.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:40.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:54:40.419 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:54:40.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:40.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:40.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:41.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:41.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:41.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:54:41.855 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:54:41.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:41.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:42.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:43.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:43.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:43.372 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:54:43.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:54:43.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:43.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:43.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:44.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:44.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:44.949 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:54:44.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:54:44.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:44.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:45.833 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:46.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:46.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:46.834 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:54:46.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:54:46.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:46.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:47.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:48.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:48.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:48.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:54:48.576 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:54:48.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:48.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:49.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:50.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:50.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:50.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:54:50.733 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:54:50.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:50.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:51.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:52.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:52.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:52.792 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:54:52.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:54:52.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:52.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:53.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:54:54.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:54:54.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:54:54.663 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:54:54.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:54:54.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:54:54.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:54:55.090 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/834257194,v1:127.0.0.1:6803/834257194] [v2:127.0.0.1:6804/834257194,v1:127.0.0.1:6805/834257194] exists,up 195d3106-9752-4c12-9ff0-98623a3e39c1 2024-09-01T02:54:55.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T02:54:55.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:54:55.096 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:54:55.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3f041d14-7d4b-4db8-8eb0-659121be134e 2024-09-01T02:54:55.098 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 3f041d14-7d4b-4db8-8eb0-659121be134e 2024-09-01T02:54:55.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3f041d14-7d4b-4db8-8eb0-659121be134e' 2024-09-01T02:54:55.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:54:55.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/19NmXjf8BhAAmnroEInDgjgT2b5OwfphHA== 2024-09-01T02:54:55.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/19NmXjf8BhAAmnroEInDgjgT2b5OwfphHA=="}' 2024-09-01T02:54:55.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3f041d14-7d4b-4db8-8eb0-659121be134e -i td/divergent-priors/1/new.json 2024-09-01T02:54:55.631 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:54:55.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-01T02:54:55.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/19NmXjf8BhAAmnroEInDgjgT2b5OwfphHA== --osd-uuid 3f041d14-7d4b-4db8-8eb0-659121be134e 2024-09-01T02:54:55.679 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:55.678+0000 7f992023e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:55.681 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:55.680+0000 7f992023e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:55.682 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:55.681+0000 7f992023e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:54:55.682 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:55.681+0000 7f992023e640 -1 bdev(0x55f8a98d7c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-01T02:54:55.682 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:54:55.681+0000 7f992023e640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-01T02:55:01.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-01T02:55:01.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:55:01.817 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T02:55:01.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T02:55:01.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:55:02.377 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T02:55:02.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T02:55:02.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:55:02.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:55:02.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:55:02.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:55:02.412 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:02.410+0000 7fd7d4d1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:02.415 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:02.414+0000 7fd7d4d1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:02.417 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:02.415+0000 7fd7d4d1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:02.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T02:55:02.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:55:02.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:55:02.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:55:02.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:55:02.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:02.823 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:55:02.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:55:02.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:02.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:03.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:03.501 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:03.500+0000 7fd7d4d1b640 -1 Falling back to public interface 2024-09-01T02:55:04.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:04.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:04.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:55:04.213 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:55:04.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:04.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:04.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:05.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:05.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:05.684 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:55:05.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:55:05.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:05.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:06.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:06.351 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:06.350+0000 7fd7d4d1b640 -1 osd.1 0 log_to_monitors true 2024-09-01T02:55:07.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:07.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:07.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:55:07.077 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:55:07.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:07.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:07.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:08.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:08.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:08.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:55:08.548 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:55:08.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:08.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:09.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:10.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:10.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:10.038 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:55:10.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:55:10.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:10.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:10.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:11.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:11.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:11.537 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:55:11.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:55:11.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:11.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:11.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:12.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:12.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:12.945 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:55:12.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:55:12.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:12.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:13.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:14.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:14.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:14.389 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:55:14.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:55:14.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:14.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:14.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:15.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:15.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:15.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:55:15.887 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:55:15.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:15.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:16.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:17.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:17.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:17.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:55:17.405 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:55:17.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:17.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:17.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:18.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:18.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:18.856 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:55:18.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:55:18.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:18.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:19.584 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:20.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:20.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:20.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:55:20.586 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:55:20.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:20.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:21.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:22.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:22.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:22.420 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:55:22.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:55:22.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:22.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:23.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:24.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:24.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:24.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:55:24.819 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:55:24.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:24.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:25.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:26.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:26.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:26.393 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:55:26.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:55:26.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:26.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:26.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:27.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:27.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:27.811 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:55:27.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:55:27.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:27.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:28.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:29.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:29.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:29.561 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:55:29.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:55:29.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:29.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:30.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:31.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:31.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:31.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:55:31.334 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T02:55:31.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:31.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:55:31.735 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1511748982,v1:127.0.0.1:6811/1511748982] [v2:127.0.0.1:6812/1511748982,v1:127.0.0.1:6813/1511748982] exists,up 3f041d14-7d4b-4db8-8eb0-659121be134e 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T02:55:31.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:55:31.740 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:55:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=71514cc4-ba30-498f-9395-a8862af51c2b 2024-09-01T02:55:31.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 71514cc4-ba30-498f-9395-a8862af51c2b' 2024-09-01T02:55:31.743 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 71514cc4-ba30-498f-9395-a8862af51c2b 2024-09-01T02:55:31.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:55:31.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCj19Nm3IRkLRAAF9RDknhwUkHInaEqMf2dmg== 2024-09-01T02:55:31.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCj19Nm3IRkLRAAF9RDknhwUkHInaEqMf2dmg=="}' 2024-09-01T02:55:31.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 71514cc4-ba30-498f-9395-a8862af51c2b -i td/divergent-priors/2/new.json 2024-09-01T02:55:32.278 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:55:32.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-01T02:55:32.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCj19Nm3IRkLRAAF9RDknhwUkHInaEqMf2dmg== --osd-uuid 71514cc4-ba30-498f-9395-a8862af51c2b 2024-09-01T02:55:32.327 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:32.324+0000 7fce2021c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:32.354 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:32.351+0000 7fce2021c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:32.355 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:32.352+0000 7fce2021c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:32.355 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:32.352+0000 7fce2021c640 -1 bdev(0x5565f78f7c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-01T02:55:32.355 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:32.352+0000 7fce2021c640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-01T02:55:37.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-01T02:55:37.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:55:37.995 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T02:55:37.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T02:55:37.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:55:38.545 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T02:55:38.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T02:55:38.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:55:38.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:55:38.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:55:38.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:55:38.582 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:38.580+0000 7f5fad127640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:38.584 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:38.583+0000 7f5fad127640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:38.585 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:38.584+0000 7f5fad127640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:55:38.981 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:55:38.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:38.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:39.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:39.425 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:39.424+0000 7f5fad127640 -1 Falling back to public interface 2024-09-01T02:55:40.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:40.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:40.407 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:55:40.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:55:40.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:40.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:40.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:41.803 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:55:41.801+0000 7f5fad127640 -1 osd.2 0 log_to_monitors true 2024-09-01T02:55:41.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:41.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:41.825 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:55:41.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:55:41.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:41.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:42.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:43.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:43.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:43.313 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:55:43.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:55:43.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:43.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:43.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:44.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:44.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:44.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:55:44.732 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:55:44.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:44.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:45.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:46.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:46.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:46.136 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:55:46.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:55:46.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:46.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:46.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:47.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:47.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:47.634 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:55:47.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:55:47.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:47.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:48.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:49.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:49.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:49.034 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:55:49.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:55:49.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:49.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:49.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:50.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:50.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:50.469 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:55:50.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:55:50.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:50.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:50.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:51.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:51.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:51.917 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:55:51.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:55:51.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:51.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:52.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:53.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:53.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:53.339 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:55:53.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:55:53.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:53.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:54.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:55.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:55.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:55.337 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:55:55.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:55:55.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:55.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:56.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:57.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:57.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:57.058 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:55:57.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:55:57.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:57.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:57.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:55:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:55:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:55:58.710 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:55:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:55:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:55:58.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:55:59.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:00.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:00.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:00.454 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:56:00.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:56:00.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:00.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:56:01.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:02.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:02.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:02.225 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:56:02.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:56:02.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:02.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:56:03.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:04.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:04.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:04.150 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:56:04.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:56:04.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:04.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:56:04.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:05.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:05.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:05.556 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:56:05.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:56:05.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:05.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:56:06.004 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2368124259,v1:127.0.0.1:6819/2368124259] [v2:127.0.0.1:6820/2368124259,v1:127.0.0.1:6821/2368124259] exists,up 71514cc4-ba30-498f-9395-a8862af51c2b 2024-09-01T02:56:06.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:56:06.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:56:06.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:56:06.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-01T02:56:06.722 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-09-01T02:56:06.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-01T02:56:07.517 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-09-01T02:56:07.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-01T02:56:08.396 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-09-01T02:56:08.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-01T02:56:08.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-01T02:56:09.243 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-09-01T02:56:09.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T02:56:10.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-01T02:56:10.859 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 size to 3 2024-09-01T02:56:10.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-01T02:56:11.521 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-09-01T02:56:11.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-01T02:56:11.532 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:56:11.532 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:56:12.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T02:56:12.044 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T02:56:12.044 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:56:12.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:56:12.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:12.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:56:12.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-01T02:56:12.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-01T02:56:12.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-01T02:56:12.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:12.164 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T02:56:12.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-01T02:56:12.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-01T02:56:12.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-09-01T02:56:12.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:12.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T02:56:12.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-01T02:56:12.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-01T02:56:12.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509446' 2024-09-01T02:56:12.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:12.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-01T02:56:12.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:12.438 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:56:12.439 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-01T02:56:12.439 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:12.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-01T02:56:12.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-01T02:56:12.440 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836500 2024-09-01T02:56:12.440 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:56:12.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-01T02:56:12.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:12.863 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-01T02:56:12.863 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:12.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T02:56:12.865 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-01T02:56:12.865 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:12.866 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-01T02:56:12.866 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672976 2024-09-01T02:56:12.866 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-01T02:56:12.866 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T02:56:13.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672976 2024-09-01T02:56:13.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:56:14.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:56:14.291 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T02:56:14.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-01T02:56:14.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:14.719 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-01T02:56:14.719 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:14.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T02:56:14.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-01T02:56:14.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-01T02:56:14.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-01T02:56:14.722 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509446 2024-09-01T02:56:14.722 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T02:56:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:56:15.115 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:56:15.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:56:15.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:56:15.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:56:15.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T02:56:15.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:56:15.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:56:15.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:56:15.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:56:15.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:56:15.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T02:56:15.620 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T02:56:15.620 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:56:15.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:56:15.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:15.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:56:15.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-01T02:56:15.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-01T02:56:15.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-01T02:56:15.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:15.740 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T02:56:15.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-01T02:56:15.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-01T02:56:15.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672978' 2024-09-01T02:56:15.860 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:15.860 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T02:56:15.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-01T02:56:15.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-01T02:56:15.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509449' 2024-09-01T02:56:15.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:15.981 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-01T02:56:15.981 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:15.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:56:15.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-01T02:56:15.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:15.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-01T02:56:15.984 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836503 2024-09-01T02:56:15.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-01T02:56:15.984 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:56:16.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-01T02:56:16.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:16.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-01T02:56:16.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:16.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T02:56:16.431 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-01T02:56:16.431 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:16.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-01T02:56:16.433 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672978 2024-09-01T02:56:16.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-01T02:56:16.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T02:56:16.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672978 2024-09-01T02:56:16.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:16.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-01T02:56:16.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:16.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T02:56:16.821 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-01T02:56:16.821 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:16.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-01T02:56:16.822 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509449 2024-09-01T02:56:16.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-01T02:56:16.822 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T02:56:17.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-01T02:56:17.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:56:17.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:56:17.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:56:17.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T02:56:17.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:56:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:56:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:56:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:56:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:56:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:56:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:56:18.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T02:56:18.216 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:56:18.216 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:56:18.216 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:56:18.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T02:56:18.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:56:18.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:56:18.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-01T02:56:18.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-01T02:56:19.121 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:56:19.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-01T02:56:19.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-01T02:56:19.133 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-09-01T02:56:19.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-01T02:56:19.501 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:56:19.501 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-01T02:56:09.803944+0000 0'0 29:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:03:19.127766+0000 0 0 2024-09-01T02:56:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:56:19.502 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:56:19.502 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:56:19.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-01T02:56:19.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-01T02:56:19.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-01T02:56:19.515 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-09-01T02:56:19.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-01T02:56:19.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:19.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-01T02:56:19.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:19.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-01T02:56:19.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:19.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-01T02:56:19.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:19.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-01T02:56:20.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-01T02:56:20.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-01T02:56:20.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-01T02:56:20.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-01T02:56:20.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-01T02:56:20.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-01T02:56:20.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:20.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-01T02:56:21.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-01T02:56:21.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-01T02:56:21.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-01T02:56:21.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-01T02:56:21.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-01T02:56:21.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-01T02:56:21.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:21.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-01T02:56:22.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-01T02:56:22.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-01T02:56:22.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-01T02:56:22.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-01T02:56:22.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-01T02:56:22.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-01T02:56:22.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:22.937 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-01T02:56:23.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-01T02:56:23.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-01T02:56:23.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-01T02:56:23.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-01T02:56:23.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-01T02:56:23.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-01T02:56:23.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:23.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-01T02:56:24.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:24.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-01T02:56:24.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:24.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-01T02:56:24.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:24.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-01T02:56:24.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:24.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-01T02:56:24.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:24.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-01T02:56:24.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:24.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-01T02:56:25.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-01T02:56:25.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-01T02:56:25.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-01T02:56:25.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-01T02:56:25.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-01T02:56:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-01T02:56:25.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:25.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-01T02:56:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:26.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-01T02:56:26.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:26.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-01T02:56:26.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:26.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-01T02:56:26.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:26.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-01T02:56:26.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:26.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-01T02:56:26.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:26.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-01T02:56:27.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-01T02:56:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-01T02:56:27.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-01T02:56:27.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-01T02:56:27.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-01T02:56:27.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-01T02:56:27.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:27.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-01T02:56:28.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-01T02:56:28.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-01T02:56:28.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-01T02:56:28.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-01T02:56:28.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-01T02:56:28.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-01T02:56:28.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:28.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-01T02:56:29.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-01T02:56:29.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-01T02:56:29.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-01T02:56:29.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-01T02:56:29.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-01T02:56:29.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-01T02:56:29.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:29.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-01T02:56:30.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-01T02:56:30.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-01T02:56:30.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-01T02:56:30.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-01T02:56:30.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-01T02:56:30.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-01T02:56:30.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:30.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-01T02:56:31.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-01T02:56:31.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-01T02:56:31.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.367 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-01T02:56:31.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-01T02:56:31.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-01T02:56:31.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-01T02:56:31.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:31.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-01T02:56:32.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-01T02:56:32.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-01T02:56:32.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-01T02:56:32.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-01T02:56:32.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-01T02:56:32.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-01T02:56:32.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:32.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-01T02:56:33.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-01T02:56:33.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-01T02:56:33.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-01T02:56:33.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-01T02:56:33.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-01T02:56:33.847 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.847 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-01T02:56:33.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-01T02:56:33.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-01T02:56:34.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T02:56:34.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-01T02:56:34.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:56:34.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:56:34.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:56:34.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T02:56:34.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:56:34.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:56:34.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:56:34.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:56:34.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:56:34.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T02:56:34.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:56:34.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:56:34.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:56:34.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:56:34.186 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:56:34.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T02:56:34.589 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T02:56:34.589 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:56:34.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:56:34.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:34.589 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:56:34.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-01T02:56:34.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-01T02:56:34.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-01T02:56:34.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:34.710 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T02:56:34.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-01T02:56:34.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-01T02:56:34.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672984' 2024-09-01T02:56:34.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:56:34.831 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T02:56:34.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-01T02:56:34.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-01T02:56:34.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672984 2-64424509454' 2024-09-01T02:56:34.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:34.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-01T02:56:34.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:34.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:56:34.952 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-01T02:56:34.953 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:34.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-01T02:56:34.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-01T02:56:34.954 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836508 2024-09-01T02:56:34.954 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:56:35.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-01T02:56:35.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:56:36.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:56:36.380 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:56:36.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836508 2024-09-01T02:56:36.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:36.814 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-01T02:56:36.814 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:36.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T02:56:36.816 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-01T02:56:36.816 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:36.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-01T02:56:36.817 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672984 2024-09-01T02:56:36.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-01T02:56:36.817 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T02:56:37.240 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672984 2024-09-01T02:56:37.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:56:37.241 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-01T02:56:37.241 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:56:37.241 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T02:56:37.242 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-01T02:56:37.242 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:56:37.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-01T02:56:37.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-01T02:56:37.244 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509454 2024-09-01T02:56:37.244 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T02:56:37.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509454 2024-09-01T02:56:37.666 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:56:37.666 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:56:37.666 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:56:38.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:56:38.584 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T02:56:38.584 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:56:38.584 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:56:38.584 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:56:39.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T02:56:39.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T02:56:39.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T02:56:39.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-01T02:56:39.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-01T02:56:39.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-01T02:56:39.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-01T02:56:39.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-01T02:56:39.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-01T02:56:39.546 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-09-01T02:56:39.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-01T02:56:39.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-01T02:56:39.913 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:56:39.913 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-01T02:56:09.803944+0000 29'100 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:03:19.127766+0000 0 0 2024-09-01T02:56:39.913 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:56:39.913 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:56:39.914 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:56:39.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-01T02:56:39.925 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-01T02:56:39.925 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-01T02:56:39.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-01T02:56:40.001 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T02:56:40.002 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T02:56:40.002 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T02:56:40.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-01T02:56:40.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-01T02:56:40.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T02:56:40.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T02:56:40.009 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:56:40.009 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:56:40.010 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:56:40.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T02:56:40.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-01T02:56:40.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-01T02:56:40.086 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T02:56:40.086 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T02:56:40.086 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T02:56:40.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-01T02:56:40.093 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-01T02:56:40.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-01T02:56:40.095 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent object 2024-09-01T02:56:40.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-01T02:56:40.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-01T02:56:40.096 INFO:tasks.workunit.client.0.smithi079.stdout:create missing divergent object 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=test 2024-09-01T02:56:40.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=existing_99 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/divergent-priors 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds test existing_99 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=test 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=existing_99 2024-09-01T02:56:40.099 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map test existing_99 2024-09-01T02:56:40.100 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-01T02:56:40.522 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-01T02:56:40.522 INFO:tasks.workunit.client.0.smithi079.stderr:0 2024-09-01T02:56:40.522 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:56:40.522 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 2024-09-01T02:56:40.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0' '2') 2024-09-01T02:56:40.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-01T02:56:40.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-01T02:56:40.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-01T02:56:40.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-01T02:56:40.523 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/divergent-priors/1/type 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:56:40.525 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:56:40.526 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T02:56:40.526 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.22149/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-01T02:56:40.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-01T02:56:40.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-01T02:56:40.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-01T02:56:40.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-01T02:56:40.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T02:56:40.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T02:56:40.611 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:56:40.611 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:56:40.612 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:56:40.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T02:56:40.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-01T02:56:40.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok injectdataerr test existing_99 2024-09-01T02:56:40.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-01T02:56:40.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-01T02:56:50.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-01T02:56:50.709 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-09-01T02:56:50.709 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-09-01T02:56:50.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-01T02:56:50.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-01T02:56:51.089 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:56:51.089 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 active+clean 2024-09-01T02:56:40.869538+0000 29'101 29:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:03:19.127766+0000 0 0 2024-09-01T02:56:51.089 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:56:51.089 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:56:51.089 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:56:51.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-01T02:56:51.100 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:56:51.100 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:56:51.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:56:51.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:56:51.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:56:51.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:56:51.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-01T02:56:51.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-01T02:56:51.696 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T02:56:51.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-01T02:56:51.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-01T02:56:52.400 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T02:56:52.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-01T02:56:52.412 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-01T02:56:52.945 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T02:56:52.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-01T02:56:52.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-01T02:56:53.455 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-09-01T02:56:53.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-01T02:56:53.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-01T02:56:53.973 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-09-01T02:56:53.984 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-09-01T02:56:53.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-01T02:56:53.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-01T02:56:54.358 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:56:54.358 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 stale+active+clean 2024-09-01T02:56:40.869538+0000 29'101 29:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:03:19.127766+0000 0 0 2024-09-01T02:56:54.358 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:56:54.358 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:56:54.358 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:56:54.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-01T02:56:54.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-01T02:56:54.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T02:56:54.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T02:56:54.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:56:54.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:56:54.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:56:54.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:56:54.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T02:56:54.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T02:56:54.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-01T02:56:54.375 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T02:56:54.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T02:56:54.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:56:54.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-01T02:56:54.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T02:56:54.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T02:56:54.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T02:56:54.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T02:56:54.407 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:54.405+0000 7f99ca14f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:56:54.424 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:54.423+0000 7f99ca14f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:56:54.425 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:54.424+0000 7f99ca14f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:56:54.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T02:56:54.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:56:54.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:56:54.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:56:54.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:56:54.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:54.796 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:56:54.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:56:54.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:54.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:56:55.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:55.518 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:55.516+0000 7f99ca14f640 -1 Falling back to public interface 2024-09-01T02:56:56.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:56.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:56.264 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:56:56.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:56:56.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:56.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:56:56.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:57.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:57.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:57.707 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:56:57.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:56:57.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:57.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:56:58.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:56:59.039 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:59.038+0000 7f99ca14f640 -1 osd.0 29 log_to_monitors true 2024-09-01T02:56:59.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:56:59.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:59.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:56:59.108 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:56:59.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:59.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:56:59.529 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1786222853,v1:127.0.0.1:6803/1786222853] [v2:127.0.0.1:6804/1786222853,v1:127.0.0.1:6805/1786222853] exists,up 195d3106-9752-4c12-9ff0-98623a3e39c1 2024-09-01T02:56:59.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:56:59.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:56:59.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:56:59.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-01T02:56:59.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-01T02:56:59.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T02:56:59.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T02:56:59.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T02:56:59.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T02:56:59.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-01T02:56:59.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:56:59.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:56:59.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:56:59.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:56:59.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T02:56:59.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T02:56:59.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T02:56:59.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T02:56:59.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:56:59.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T02:56:59.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:56:59.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:56:59.541 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:56:59.541 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:56:59.541 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:56:59.541 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:56:59.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:56:59.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T02:56:59.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T02:56:59.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-01T02:56:59.543 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T02:56:59.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T02:56:59.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:56:59.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-01T02:56:59.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T02:56:59.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T02:56:59.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T02:56:59.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T02:56:59.566 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:59.564+0000 7f8cf040e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:56:59.586 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:59.585+0000 7f8cf040e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:56:59.587 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:56:59.586+0000 7f8cf040e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:56:59.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T02:56:59.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:56:59.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T02:56:59.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:56:59.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:56:59.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:56:59.924 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:56:59.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:56:59.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:56:59.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:57:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:57:01.171 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:01.170+0000 7f8cf040e640 -1 Falling back to public interface 2024-09-01T02:57:01.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:57:01.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:01.316 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:57:01.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:57:01.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:01.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:57:01.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:57:02.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:57:02.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:02.752 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:57:02.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:57:02.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:02.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:57:03.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:57:04.156 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:04.155+0000 7f8cf040e640 -1 osd.2 29 log_to_monitors true 2024-09-01T02:57:04.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:57:04.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:04.197 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:57:04.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:57:04.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:04.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:57:04.633 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1055510456,v1:127.0.0.1:6811/1055510456] [v2:127.0.0.1:6812/1055510456,v1:127.0.0.1:6813/1055510456] exists,up 71514cc4-ba30-498f-9395-a8862af51c2b 2024-09-01T02:57:04.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:57:04.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:57:04.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:57:04.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-01T02:57:04.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-01T02:57:05.224 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-09-01T02:57:05.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-01T02:57:05.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-01T02:57:05.789 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-09-01T02:57:05.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T02:57:05.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-01T02:57:05.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T02:57:05.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T02:57:05.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T02:57:05.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T02:57:05.802 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T02:57:05.802 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T02:57:05.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T02:57:05.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T02:57:05.803 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T02:57:05.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T02:57:05.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T02:57:05.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T02:57:05.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T02:57:05.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T02:57:05.855 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T02:57:06.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T02:57:06.384 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T02:57:06.385 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:57:06.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T02:57:06.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:57:06.385 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T02:57:06.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-01T02:57:06.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-01T02:57:06.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-01T02:57:06.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:57:06.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T02:57:06.645 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T02:57:06.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T02:57:06.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T02:57:06.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T02:57:06.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T02:57:06.647 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T02:57:06.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-01T02:57:06.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-01T02:57:06.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-01T02:57:06.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:57:06.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-01T02:57:06.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:57:06.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T02:57:06.768 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-01T02:57:06.768 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:57:06.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-01T02:57:06.770 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 154618822660 2024-09-01T02:57:06.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-01T02:57:06.770 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:57:07.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-01T02:57:07.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T02:57:08.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T02:57:08.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T02:57:08.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-01T02:57:08.657 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T02:57:08.657 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-01T02:57:08.657 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T02:57:08.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T02:57:08.658 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-01T02:57:08.658 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T02:57:08.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-01T02:57:08.659 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 163208757251 2024-09-01T02:57:08.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-01T02:57:08.660 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T02:57:09.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-01T02:57:09.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T02:57:09.092 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:09.092 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:09.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:10.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:10.023 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:10.023 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:10.023 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:10.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:10.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T02:57:10.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T02:57:10.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T02:57:10.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:10.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:10.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:11.032 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:11.032 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:11.033 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:11.033 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:11.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:11.597 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:12.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:12.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:12.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T02:57:12.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:12.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:12.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:12.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:12.720 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:12.720 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:12.720 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:13.239 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:13.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:13.240 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:13.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:13.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:13.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T02:57:13.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:13.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T02:57:14.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:14.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:14.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:14.577 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:14.578 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:14.578 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:15.114 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:15.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:15.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:15.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T02:57:15.661 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:15.662 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:16.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:16.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:16.845 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:16.846 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:16.846 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:17.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:17.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:17.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:17.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:17.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:17.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:17.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:17.370 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:17.370 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:17.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:17.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:17.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T02:57:17.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:17.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T02:57:19.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:19.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:19.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:19.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:19.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:19.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:19.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:19.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:19.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:19.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:19.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:19.918 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:20.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:20.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:20.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:20.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:20.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T02:57:20.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:20.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T02:57:24.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:24.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:24.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:24.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:24.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:24.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:24.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:24.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:24.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:24.587 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:24.588 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:24.588 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:25.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:25.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:25.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:25.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:25.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:25.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:25.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:25.112 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:25.112 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:25.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:25.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:25.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T02:57:25.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:25.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T02:57:32.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:32.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:32.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:32.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:32.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:32.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:32.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:32.056 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:32.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:32.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:32.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:32.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:32.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:32.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:32.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:32.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:32.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:32.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:32.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:32.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:32.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:33.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:33.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:33.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T02:57:33.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T02:57:33.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T02:57:40.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T02:57:40.883 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T02:57:41.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T02:57:41.273 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T02:57:41.273 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T02:57:41.273 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T02:57:41.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T02:57:41.794 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T02:57:42.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T02:57:42.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T02:57:42.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T02:57:42.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "d43c603c-d5ba-41cc-b087-f0bdbf9ca6e3", 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T02:57:42.869977+0000", 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T02:57:42.872 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T02:57:42.873 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T02:57:42.874 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T02:57:42.875 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T02:57:42.876 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T02:57:42.877 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T02:54:18.478225Z", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T02:54:18.478225Z", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T02:57:42.878 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T02:57:42.879 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.880 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 42, 2024-09-01T02:57:42.881 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T02:54:19.429842+0000", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T02:57:05.895829+0000", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T02:57:04.360156+0000", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T02:57:05.581459+0000", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T02:57:42.882 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T02:57:42.883 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T02:56:08.726476+0000", 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T02:57:42.884 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-09-01T02:57:42.885 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "29", 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T02:57:42.886 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T02:57:42.887 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T02:57:42.888 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.9900000095367432, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.9900000095367432, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-09-01T02:57:42.889 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 2, 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 2, 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "195d3106-9752-4c12-9ff0-98623a3e39c1", 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T02:57:42.890 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 36, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 41, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 30, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.891 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.892 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.893 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.894 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.895 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1786222853 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/1786222853", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/1786222853", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1786222853", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1786222853", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T02:57:42.896 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "3f041d14-7d4b-4db8-8eb0-659121be134e", 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 23, 2024-09-01T02:57:42.897 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.898 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.899 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T02:57:42.900 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.901 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1511748982 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1511748982", 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1511748982", 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1511748982", 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1511748982", 2024-09-01T02:57:42.902 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "71514cc4-ba30-498f-9395-a8862af51c2b", 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T02:57:42.903 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 38, 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T02:57:42.904 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.905 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.906 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.907 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1055510456 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1055510456", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1055510456", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1055510456", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1055510456", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T02:57:42.908 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T02:56:51.532911+0000", 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:54:53.623043+0000", 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T02:57:42.909 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T02:56:52.076459+0000", 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:55:29.987354+0000", 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T02:57:42.910 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T02:56:52.771464+0000", 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:56:04.337653+0000", 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T02:57:42.911 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T02:57:42.912 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.913 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/1786222853,v1:127.0.0.1:6805/1786222853]", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T02:57:42.914 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T02:57:42.915 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:54:25.780847Z", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/1786222853,v1:127.0.0.1:6803/1786222853]", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1786222853,v1:127.0.0.1:6809/1786222853]", 2024-09-01T02:57:42.916 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1786222853,v1:127.0.0.1:6807/1786222853]", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T02:57:42.917 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1511748982,v1:127.0.0.1:6813/1511748982]", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T02:57:42.918 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T02:57:42.919 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:55:01.179941Z", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1511748982,v1:127.0.0.1:6811/1511748982]", 2024-09-01T02:57:42.920 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1511748982,v1:127.0.0.1:6817/1511748982]", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1511748982,v1:127.0.0.1:6815/1511748982]", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T02:57:42.921 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1055510456,v1:127.0.0.1:6813/1055510456]", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T02:57:42.922 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T02:57:42.923 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:55:37.346196Z", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1055510456,v1:127.0.0.1:6811/1055510456]", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1055510456,v1:127.0.0.1:6817/1055510456]", 2024-09-01T02:57:42.924 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1055510456,v1:127.0.0.1:6815/1055510456]", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T02:57:42.925 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 36 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.926 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 39 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 42, 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T02:57:42.927 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T02:57:42.928 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T02:57:42.929 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.930 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.931 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T02:57:42.932 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T02:57:42.933 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.934 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T02:57:42.935 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.936 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T02:57:42.937 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.938 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T02:57:42.939 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T02:57:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T02:57:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T02:57:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T02:54:19:405336+0000", 2024-09-01T02:57:42.943 INFO:tasks.workunit.client.0.smithi079.stderr:report 2625978186 2024-09-01T02:57:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T02:57:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T02:57:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T02:57:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T02:57:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T02:57:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48300, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T02:57:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T02:57:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T02:57:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T02:57:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-09-01T02:57:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83688, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2128, 2024-09-01T02:57:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489112, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036850688, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2179072, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 425532, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T02:57:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 1, 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T02:57:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T02:57:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83688, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2128, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489112, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036850688, 2024-09-01T02:57:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2179072, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 425532, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 1, 2024-09-01T02:57:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48300, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T02:57:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T02:57:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T02:57:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T02:57:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1228800, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 144900, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T02:57:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 154618822667 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672988 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T02:57:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 163208757258 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T02:57:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T02:57:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 4, 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T02:56:32.197239+0000", 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "services": { 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": { 2024-09-01T02:57:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "daemons": { 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": "", 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "2": { 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "start_epoch": 0, 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "start_stamp": "0.000000", 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "gid": 0, 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "metadata": {}, 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "task_status": {} 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T02:57:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 91, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 382, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 523, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T02:57:42.968 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T02:57:42.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T02:57:42.969 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-01T02:57:42.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-01T02:57:42.969 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-09-01T02:57:42.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-01T02:57:42.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:43.274 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:43.274 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 29'100 42:132 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:43.274 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:43.274 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:43.275 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:43.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-01T02:57:43.402 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-09-01T02:57:43.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-01T02:57:43.402 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:43.764 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:43.764 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 29'100 42:132 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:43.764 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:43.764 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:43.765 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:43.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-01T02:57:43.776 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-01T02:57:43.776 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-01T02:57:43.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok set_recovery_delay 100000 2024-09-01T02:57:43.857 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T02:57:43.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-01T02:57:43.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok set_recovery_delay 100000 2024-09-01T02:57:43.945 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T02:57:43.952 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-09-01T02:57:43.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-01T02:57:43.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:44.325 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:44.326 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 42'101 42:133 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:44.326 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:44.326 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:44.326 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:44.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-01T02:57:44.977 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-09-01T02:57:44.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-01T02:57:44.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T02:57:44.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:57:44.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T02:57:44.995 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T02:57:44.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T02:57:44.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:57:44.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T02:57:44.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T02:57:44.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T02:57:44.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T02:57:44.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T02:57:45.026 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:45.024+0000 7fc932a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:57:45.045 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:45.043+0000 7fc932a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:57:45.046 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:45.044+0000 7fc932a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:57:45.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-01T02:57:45.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-01T02:57:46.141 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:46.139+0000 7fc932a12640 -1 Falling back to public interface 2024-09-01T02:57:49.158 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:49.156+0000 7fc932a12640 -1 osd.1 29 log_to_monitors true 2024-09-01T02:57:50.495 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-09-01T02:57:50.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-01T02:57:50.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:50.881 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:50.881 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 42'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:50.881 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:50.881 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:50.881 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-01T02:57:50.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok set_recovery_delay 100000 2024-09-01T02:57:50.973 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T02:57:50.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-01T02:57:51.586 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-09-01T02:57:51.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-01T02:57:51.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:57:51.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:57:51.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:57:51.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:57:51.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:51.598 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:57:51.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:57:51.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:51.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:57:52.004 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1786222853,v1:127.0.0.1:6803/1786222853] [v2:127.0.0.1:6804/1786222853,v1:127.0.0.1:6805/1786222853] exists,up 195d3106-9752-4c12-9ff0-98623a3e39c1 2024-09-01T02:57:52.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:57:52.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:57:52.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:57:52.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-01T02:57:52.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:57:52.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:57:52.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:57:52.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:57:52.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:52.006 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:57:52.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:57:52.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:52.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:57:52.425 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2946551816,v1:127.0.0.1:6819/2946551816] [v2:127.0.0.1:6820/2946551816,v1:127.0.0.1:6821/2946551816] exists,up 3f041d14-7d4b-4db8-8eb0-659121be134e 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:57:52.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:52.428 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:57:52.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:57:52.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:52.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:57:52.815 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1055510456,v1:127.0.0.1:6811/1055510456] [v2:127.0.0.1:6812/1055510456,v1:127.0.0.1:6813/1055510456] exists,up 71514cc4-ba30-498f-9395-a8862af51c2b 2024-09-01T02:57:52.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:57:52.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:57:52.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:57:52.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:53.188 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:53.189 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 42'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:53.189 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:53.189 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:53.189 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:53.200 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-09-01T02:57:53.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-01T02:57:53.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:53.569 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:53.569 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 42'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:53.570 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:53.570 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:53.570 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:53.581 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-01T02:57:53.686 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-09-01T02:57:53.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-01T02:57:53.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:54.057 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:54.057 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 42'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:54.058 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:54.058 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:54.058 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:54.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-01T02:57:54.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:57:54.069 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:57:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:57:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:57:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:57:54.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:57:54.174 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-09-01T02:57:54.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-01T02:57:54.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-01T02:57:54.543 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:57:54.543 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T02:57:06.187815+0000 42'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:48:09.348054+0000 0 0 2024-09-01T02:57:54.543 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:57:54.544 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:57:54.544 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:57:54.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-01T02:57:54.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T02:57:54.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:57:54.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:57:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:57:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T02:57:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T02:57:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T02:57:54.560 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T02:57:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T02:57:54.592 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:54.590+0000 7fc03f2dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:57:54.594 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:54.592+0000 7fc03f2dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:57:54.595 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:54.593+0000 7fc03f2dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:57:54.947 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:57:54.948 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:57:54.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:57:54.948 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:57:55.367 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2946551816,v1:127.0.0.1:6819/2946551816] [v2:127.0.0.1:6820/2946551816,v1:127.0.0.1:6821/2946551816] exists,up 3f041d14-7d4b-4db8-8eb0-659121be134e 2024-09-01T02:57:55.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:57:55.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:57:55.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:57:55.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-01T02:57:55.940 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:55.938+0000 7fc03f2dc640 -1 Falling back to public interface 2024-09-01T02:57:58.895 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:57:58.894+0000 7fc03f2dc640 -1 osd.1 48 log_to_monitors true 2024-09-01T02:58:15.369 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-09-01T02:58:15.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-01T02:58:15.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-01T02:58:15.747 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:58:15.748 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-01T02:58:00.661385+0000 48'102 52:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:24:58.489181+0000 0 0 2024-09-01T02:58:15.748 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:58:15.748 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:58:15.748 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:58:15.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-01T02:58:15.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-01T02:58:15.869 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T02:58:15.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-01T02:58:15.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-01T02:58:15.991 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T02:58:15.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-01T02:58:16.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-01T02:58:16.111 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T02:58:16.119 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-09-01T02:58:16.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-01T02:58:16.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-01T02:58:16.490 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T02:58:16.491 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-01T02:58:00.661385+0000 48'102 52:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T02:56:08.780432+0000 0'0 2024-09-01T02:56:08.780432+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:24:58.489181+0000 0 0 2024-09-01T02:58:16.491 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T02:58:16.491 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T02:58:16.491 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T02:58:16.502 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-01T02:58:16.503 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-01T02:58:16.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T02:58:16.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-01T02:58:16.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T02:58:16.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-01T02:58:16.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T02:58:16.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-01T02:58:16.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T02:58:16.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-01T02:58:16.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-01T02:58:16.665 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-01T02:58:16.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T02:58:16.671 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-09-01T02:56:40.127895+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T02:58:16.671 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-09-01T02:56:40.127895+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T02:58:16.671 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-09-01T02:56:40.127895+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T02:57:52.674+0000 7fc916281640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-01T02:58:16.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-01T02:58:16.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T02:58:16.677 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-09-01T02:58:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-01T02:58:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-01T02:58:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-01T02:58:16.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-01T02:58:17.165 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-09-01T02:58:17.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-01T02:58:17.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:58:17.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:58:17.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:58:17.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:58:17.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:58:17.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:58:17.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:58:17.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:58:17.289 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:58:17.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:58:17.290 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:58:17.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:58:17.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:58:17.292 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:58:17.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:58:17.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:58:17.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:58:17.293 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:58:17.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:58:17.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:58:17.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T02:58:17.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:58:17.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:17.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:17.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T02:58:17.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:58:17.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:58:17.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-01T02:58:17.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-01T02:58:17.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-01T02:58:17.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T02:58:17.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T02:58:17.318 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T02:58:17.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T02:58:17.318 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T02:58:17.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T02:58:17.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T02:58:17.320 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T02:58:17.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:58:17.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T02:58:17.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T02:58:17.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T02:58:17.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T02:58:17.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T02:58:17.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T02:58:17.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T02:58:17.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:17.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:17.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T02:58:17.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T02:58:17.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T02:58:17.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-01T02:58:17.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T02:58:17.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:17.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:17.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.22149 2024-09-01T02:58:17.328 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T02:58:17.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T02:58:17.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-01T02:58:17.329 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-01T02:58:17.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-01T02:58:17.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:58:17.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T02:58:17.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T02:58:17.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T02:58:17.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:58:17.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get fsid 2024-09-01T02:58:18.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T02:58:18.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T02:58:18.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T02:58:18.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T02:58:18.074 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T02:58:18.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get mon_host 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-01T02:58:18.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T02:58:19.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:58:19.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T02:58:19.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-01T02:58:19.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-01T02:58:19.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T02:58:19.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:58:19.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T02:58:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:58:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-01T02:58:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:58:19.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:58:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:58:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:58:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T02:58:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T02:58:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:58:19.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:19.048 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:19.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:58:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:58:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:58:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:58:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:58:19.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:58:19.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:58:19.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:58:19.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:58:19.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:58:19.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:58:19.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:58:19.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:58:19.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-01T02:58:19.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:58:19.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b494a832-c285-435b-9e09-53bb38fafb41 2024-09-01T02:58:19.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b494a832-c285-435b-9e09-53bb38fafb41' 2024-09-01T02:58:19.052 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 b494a832-c285-435b-9e09-53bb38fafb41 2024-09-01T02:58:19.052 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:58:19.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBL2NNmss7zAxAAINxzh+JACKxJxyvCx00SXw== 2024-09-01T02:58:19.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBL2NNmss7zAxAAINxzh+JACKxJxyvCx00SXw=="}' 2024-09-01T02:58:19.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b494a832-c285-435b-9e09-53bb38fafb41 -i td/divergent-priors/0/new.json 2024-09-01T02:58:19.327 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:58:19.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-01T02:58:19.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBL2NNmss7zAxAAINxzh+JACKxJxyvCx00SXw== --osd-uuid b494a832-c285-435b-9e09-53bb38fafb41 2024-09-01T02:58:19.375 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:19.372+0000 7f114c29c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:19.388 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:19.385+0000 7f114c29c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:19.389 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:19.386+0000 7f114c29c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:19.389 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:19.386+0000 7f114c29c640 -1 bdev(0x55f959b43c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-01T02:58:19.389 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:19.387+0000 7f114c29c640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-01T02:58:25.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-01T02:58:25.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:58:25.365 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T02:58:25.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T02:58:25.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:58:25.905 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T02:58:25.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T02:58:25.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:58:25.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:58:25.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:58:25.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:58:25.940 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:25.938+0000 7f613c015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:25.961 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:25.960+0000 7f613c015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:25.962 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:25.961+0000 7f613c015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:26.330 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:58:26.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:58:26.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:26.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:26.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:27.317 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:27.316+0000 7f613c015640 -1 Falling back to public interface 2024-09-01T02:58:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:27.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:58:27.761 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:58:27.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:27.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:28.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:29.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:29.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:29.195 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:58:29.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:58:29.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:29.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:29.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:30.239 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:30.238+0000 7f613c015640 -1 osd.0 0 log_to_monitors true 2024-09-01T02:58:30.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:30.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:30.638 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:58:30.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:58:30.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:30.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:31.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:32.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:32.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:32.035 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:58:32.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:58:32.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:32.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:32.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:33.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:33.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:33.466 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:58:33.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:58:33.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:33.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:33.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:34.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:34.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:34.863 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:58:34.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:58:34.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:34.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:35.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:36.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:36.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:36.355 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:58:36.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:58:36.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:36.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:36.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:37.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:37.769 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:58:37.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:37.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:58:37.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:37.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:38.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:39.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:39.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:39.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:58:39.176 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:58:39.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:39.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:39.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:40.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:40.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:40.577 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:58:40.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:58:40.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:40.577 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:41.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:42.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:42.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:42.023 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:58:42.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:58:42.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:42.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:42.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:43.593 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:58:43.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:58:43.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:43.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:44.369 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:45.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:45.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:45.371 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:58:45.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:58:45.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:45.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:46.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:47.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:47.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:47.098 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:58:47.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:58:47.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:47.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:47.608 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:48.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:48.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:48.610 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:58:48.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:58:48.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:48.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:49.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:50.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:50.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:50.318 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:58:50.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:58:50.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:50.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:51.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:52.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:52.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:52.500 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:58:52.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:58:52.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:52.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:53.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:58:54.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:58:54.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:58:54.116 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T02:58:54.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:58:54.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:58:54.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T02:58:54.513 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2509930328,v1:127.0.0.1:6803/2509930328] [v2:127.0.0.1:6804/2509930328,v1:127.0.0.1:6805/2509930328] exists,up b494a832-c285-435b-9e09-53bb38fafb41 2024-09-01T02:58:54.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:58:54.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:58:54.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:58:54.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-01T02:58:54.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-01T02:58:54.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:58:54.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:58:54.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 690f6dfa-5c78-4412-94fc-a2759be4428d' 2024-09-01T02:58:54.521 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T02:58:54.522 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:58:54.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBu2NNm6RUsIBAAmuH40hmsTpSzZalHJmPAcA== 2024-09-01T02:58:54.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBu2NNm6RUsIBAAmuH40hmsTpSzZalHJmPAcA=="}' 2024-09-01T02:58:54.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 690f6dfa-5c78-4412-94fc-a2759be4428d -i td/divergent-priors/1/new.json 2024-09-01T02:58:54.995 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:58:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-01T02:58:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBu2NNm6RUsIBAAmuH40hmsTpSzZalHJmPAcA== --osd-uuid 690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T02:58:55.044 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:55.042+0000 7f9e71c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:55.063 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:55.062+0000 7f9e71c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:55.064 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:55.063+0000 7f9e71c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:58:55.065 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:55.063+0000 7f9e71c13640 -1 bdev(0x562d3ffa3c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-01T02:58:55.065 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:58:55.063+0000 7f9e71c13640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-01T02:59:00.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-01T02:59:00.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:59:00.740 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T02:59:00.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T02:59:00.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:59:01.282 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T02:59:01.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T02:59:01.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:59:01.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:59:01.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:59:01.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:59:01.318 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:01.315+0000 7f5141a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:01.329 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:01.327+0000 7f5141a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:01.330 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:01.328+0000 7f5141a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:59:01.717 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:59:01.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:01.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:02.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:02.915 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:02.912+0000 7f5141a0f640 -1 Falling back to public interface 2024-09-01T02:59:03.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:03.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:03.108 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:59:03.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:59:03.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:03.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:03.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:04.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:04.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:04.587 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:59:04.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:59:04.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:04.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:05.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:05.774 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:05.772+0000 7f5141a0f640 -1 osd.1 0 log_to_monitors true 2024-09-01T02:59:06.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:06.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:06.071 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:59:06.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:59:06.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:06.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:06.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:07.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:07.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:07.616 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:59:07.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:59:07.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:07.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:08.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:09.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:09.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:09.018 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:59:09.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:59:09.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:09.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:09.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:10.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:10.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:10.549 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:59:10.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:59:10.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:10.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:10.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:11.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:11.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:11.991 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:59:11.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:59:11.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:11.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:12.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:13.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:13.390 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:13.391 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:59:13.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:59:13.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:13.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:13.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:14.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:14.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:14.829 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:59:14.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:59:14.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:14.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:15.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:16.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:16.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:16.244 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:59:16.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:59:16.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:16.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:16.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:17.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:17.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:17.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:59:17.776 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:59:17.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:17.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:18.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:19.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:19.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:19.180 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:59:19.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:59:19.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:19.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:20.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:21.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:21.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:21.263 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:59:21.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:59:21.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:21.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:21.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:22.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:22.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:22.983 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:59:22.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:59:22.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:22.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:23.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:24.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:24.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:24.841 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T02:59:24.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T02:59:24.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:24.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:25.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:26.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:26.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:26.628 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T02:59:26.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T02:59:26.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:26.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:27.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:28.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:28.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:28.230 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T02:59:28.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T02:59:28.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:28.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:28.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:29.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:29.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:29.792 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T02:59:29.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T02:59:29.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:29.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T02:59:30.219 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/920365757,v1:127.0.0.1:6811/920365757] [v2:127.0.0.1:6812/920365757,v1:127.0.0.1:6813/920365757] exists,up 690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T02:59:30.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T02:59:30.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T02:59:30.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:59:30.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ecd3a08c-2f13-4e17-9be6-31763cf0d625 2024-09-01T02:59:30.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ecd3a08c-2f13-4e17-9be6-31763cf0d625' 2024-09-01T02:59:30.227 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 ecd3a08c-2f13-4e17-9be6-31763cf0d625 2024-09-01T02:59:30.227 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T02:59:30.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCS2NNmAfqjDhAA3iw38CJli43VAn7Ocf1PWQ== 2024-09-01T02:59:30.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCS2NNmAfqjDhAA3iw38CJli43VAn7Ocf1PWQ=="}' 2024-09-01T02:59:30.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ecd3a08c-2f13-4e17-9be6-31763cf0d625 -i td/divergent-priors/2/new.json 2024-09-01T02:59:30.746 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:59:30.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-01T02:59:30.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCS2NNmAfqjDhAA3iw38CJli43VAn7Ocf1PWQ== --osd-uuid ecd3a08c-2f13-4e17-9be6-31763cf0d625 2024-09-01T02:59:30.796 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:30.795+0000 7f8fd420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:30.805 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:30.803+0000 7f8fd420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:30.806 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:30.804+0000 7f8fd420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:30.806 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:30.805+0000 7f8fd420f640 -1 bdev(0x55e524a11c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-01T02:59:30.806 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:30.805+0000 7f8fd420f640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-01T02:59:36.915 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-01T02:59:36.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T02:59:36.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T02:59:36.916 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T02:59:36.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T02:59:37.481 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T02:59:37.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T02:59:37.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T02:59:37.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T02:59:37.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T02:59:37.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T02:59:37.516 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:37.515+0000 7f2e3f61b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:37.521 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:37.519+0000 7f2e3f61b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:37.522 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:37.520+0000 7f2e3f61b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T02:59:37.935 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T02:59:37.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:37.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:38.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:39.117 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:39.116+0000 7f2e3f61b640 -1 Falling back to public interface 2024-09-01T02:59:39.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:39.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:39.362 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T02:59:39.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T02:59:39.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:39.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:39.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:40.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:40.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:40.789 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T02:59:40.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T02:59:40.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:40.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:41.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:41.809 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T02:59:41.806+0000 7f2e3f61b640 -1 osd.2 0 log_to_monitors true 2024-09-01T02:59:42.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:42.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:42.215 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T02:59:42.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T02:59:42.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:42.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:42.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:43.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:43.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:43.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T02:59:43.624 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T02:59:43.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:43.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:44.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:45.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:45.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:45.121 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T02:59:45.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T02:59:45.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:45.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:45.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:46.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:46.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:46.557 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T02:59:46.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T02:59:46.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:46.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:46.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:47.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:47.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:47.959 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T02:59:47.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T02:59:47.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:47.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:48.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:49.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:49.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:49.360 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T02:59:49.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T02:59:49.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:49.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:49.847 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:50.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:50.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:50.850 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T02:59:50.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T02:59:50.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:50.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:51.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:52.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:52.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:52.251 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T02:59:52.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T02:59:52.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:52.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:52.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:53.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:53.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:53.789 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T02:59:53.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T02:59:53.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:53.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:54.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:55.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:55.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:55.387 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T02:59:55.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T02:59:55.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:55.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:56.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:57.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:57.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:57.318 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T02:59:57.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T02:59:57.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:57.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T02:59:58.153 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T02:59:59.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T02:59:59.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T02:59:59.155 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T02:59:59.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T02:59:59.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T02:59:59.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:00:00.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:00:01.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:00:01.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:01.079 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:00:01.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:00:01.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:01.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:00:02.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:00:03.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:00:03.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:03.477 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:00:03.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:00:03.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:03.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:00:04.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:00:05.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:00:05.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:05.488 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:00:05.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:00:05.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:05.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:00:05.931 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2956044914,v1:127.0.0.1:6819/2956044914] [v2:127.0.0.1:6820/2956044914,v1:127.0.0.1:6821/2956044914] exists,up ecd3a08c-2f13-4e17-9be6-31763cf0d625 2024-09-01T03:00:05.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:00:05.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:00:05.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:00:05.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-01T03:00:06.622 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-09-01T03:00:06.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-01T03:00:07.360 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-09-01T03:00:07.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-01T03:00:08.148 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-09-01T03:00:08.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-01T03:00:08.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-01T03:00:08.755 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-09-01T03:00:08.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T03:00:09.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-01T03:00:10.385 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 size to 3 2024-09-01T03:00:10.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-01T03:00:11.064 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-09-01T03:00:11.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-01T03:00:11.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:00:11.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:00:11.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:00:11.525 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:00:11.525 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:00:11.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:00:11.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:11.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:00:11.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-01T03:00:11.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-01T03:00:11.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-01T03:00:11.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:11.664 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:00:11.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-01T03:00:11.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-01T03:00:11.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672973' 2024-09-01T03:00:11.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:11.784 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:00:11.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-01T03:00:11.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-01T03:00:11.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672973 2-64424509452' 2024-09-01T03:00:11.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:11.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-01T03:00:11.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:11.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:00:11.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-01T03:00:11.907 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:11.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-01T03:00:11.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-01T03:00:11.909 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836503 2024-09-01T03:00:11.909 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:00:12.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-01T03:00:12.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:00:13.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:00:13.295 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:00:13.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-01T03:00:13.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:13.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-01T03:00:13.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:13.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:00:13.735 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-01T03:00:13.735 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:13.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-01T03:00:13.736 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672973 2024-09-01T03:00:13.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-01T03:00:13.737 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:00:14.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-01T03:00:14.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:14.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-01T03:00:14.169 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:14.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:00:14.170 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-01T03:00:14.170 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:14.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-01T03:00:14.171 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509452 2024-09-01T03:00:14.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-01T03:00:14.171 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:00:14.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-01T03:00:14.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-01T03:00:14.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:00:14.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:00:14.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:00:14.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:00:14.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:00:14.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:00:14.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:00:14.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:00:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:00:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:00:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:00:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:00:15.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:15.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:00:15.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-01T03:00:15.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-01T03:00:15.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-01T03:00:15.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:15.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:00:15.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-01T03:00:15.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-01T03:00:15.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672976' 2024-09-01T03:00:15.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:15.416 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:00:15.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-01T03:00:15.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-01T03:00:15.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672976 2-64424509455' 2024-09-01T03:00:15.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:15.536 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-01T03:00:15.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:15.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:00:15.538 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-01T03:00:15.538 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:15.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-01T03:00:15.539 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836506 2024-09-01T03:00:15.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-01T03:00:15.539 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:00:15.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-01T03:00:15.926 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:00:16.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:00:16.927 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:00:17.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-01T03:00:17.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:17.374 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-01T03:00:17.375 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:17.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:00:17.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-01T03:00:17.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:17.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-01T03:00:17.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-01T03:00:17.377 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672976 2024-09-01T03:00:17.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:00:17.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-01T03:00:17.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:17.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-01T03:00:17.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:17.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:00:17.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-01T03:00:17.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:17.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-01T03:00:17.820 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509455 2024-09-01T03:00:17.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-01T03:00:17.821 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:00:18.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-01T03:00:18.294 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:00:18.294 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:00:18.295 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:00:18.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:00:19.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:00:19.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:00:19.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:00:19.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:00:19.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:00:19.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:00:19.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:00:19.752 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-01T03:00:19.752 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-01T03:00:20.145 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:00:20.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-01T03:00:20.156 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-09-01T03:00:20.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-01T03:00:20.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:00:20.528 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:00:20.528 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-01T03:00:09.715177+0000 0'0 29:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:19:33.434166+0000 0 0 2024-09-01T03:00:20.528 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:00:20.528 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:00:20.528 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:00:20.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-01T03:00:20.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-01T03:00:20.541 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-09-01T03:00:20.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-01T03:00:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:20.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-01T03:00:20.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:20.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-01T03:00:20.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:20.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-01T03:00:21.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-01T03:00:21.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-01T03:00:21.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-01T03:00:21.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-01T03:00:21.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-01T03:00:21.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-01T03:00:21.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:21.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-01T03:00:22.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-01T03:00:22.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-01T03:00:22.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-01T03:00:22.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-01T03:00:22.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-01T03:00:22.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-01T03:00:22.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:22.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-01T03:00:23.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:23.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-01T03:00:23.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:23.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-01T03:00:23.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:23.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-01T03:00:23.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:23.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-01T03:00:23.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:23.715 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-01T03:00:23.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:23.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-01T03:00:24.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-01T03:00:24.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-01T03:00:24.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-01T03:00:24.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-01T03:00:24.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-01T03:00:24.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-01T03:00:24.904 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:24.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-01T03:00:25.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-01T03:00:25.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-01T03:00:25.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-01T03:00:25.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-01T03:00:25.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-01T03:00:25.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-01T03:00:25.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:25.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-01T03:00:26.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-01T03:00:26.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-01T03:00:26.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-01T03:00:26.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-01T03:00:26.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-01T03:00:26.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-01T03:00:26.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:26.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-01T03:00:27.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:27.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-01T03:00:27.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:27.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-01T03:00:27.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:27.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-01T03:00:27.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:27.579 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-01T03:00:27.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:27.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-01T03:00:27.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:27.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-01T03:00:28.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-01T03:00:28.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-01T03:00:28.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-01T03:00:28.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-01T03:00:28.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-01T03:00:28.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-01T03:00:28.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:28.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-01T03:00:29.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-01T03:00:29.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-01T03:00:29.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-01T03:00:29.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-01T03:00:29.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-01T03:00:29.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-01T03:00:29.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:29.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-01T03:00:30.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:30.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-01T03:00:30.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:30.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-01T03:00:30.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:30.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-01T03:00:30.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:30.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-01T03:00:30.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:30.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-01T03:00:30.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:30.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-01T03:00:31.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-01T03:00:31.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-01T03:00:31.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-01T03:00:31.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-01T03:00:31.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-01T03:00:31.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-01T03:00:31.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:31.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-01T03:00:32.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:32.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-01T03:00:32.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:32.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-01T03:00:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:32.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-01T03:00:32.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:32.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-01T03:00:32.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:32.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-01T03:00:32.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:32.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-01T03:00:33.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-01T03:00:33.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-01T03:00:33.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-01T03:00:33.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-01T03:00:33.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-01T03:00:33.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-01T03:00:33.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:33.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-01T03:00:34.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-01T03:00:34.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-01T03:00:34.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-01T03:00:34.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-01T03:00:34.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-01T03:00:34.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-01T03:00:34.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:34.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-01T03:00:35.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:35.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-01T03:00:35.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:35.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-01T03:00:35.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-01T03:00:35.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:00:35.645 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:00:35.646 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:00:35.646 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:00:35.646 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:00:35.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:00:35.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:00:35.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:00:35.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:00:35.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:00:35.699 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:00:36.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:00:36.122 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:00:36.122 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:00:36.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:00:36.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:36.123 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:00:36.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-01T03:00:36.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-01T03:00:36.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-01T03:00:36.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:36.244 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:00:36.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-01T03:00:36.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-01T03:00:36.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672982' 2024-09-01T03:00:36.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:00:36.364 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:00:36.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-01T03:00:36.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-01T03:00:36.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672982 2-64424509461' 2024-09-01T03:00:36.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:36.484 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-01T03:00:36.484 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:36.484 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:00:36.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-01T03:00:36.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:36.486 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-01T03:00:36.486 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836512 2024-09-01T03:00:36.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-01T03:00:36.487 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:00:36.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836512 2024-09-01T03:00:36.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:00:37.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:00:37.916 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:00:38.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-01T03:00:38.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:38.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-01T03:00:38.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:38.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:00:38.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-01T03:00:38.315 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:38.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-01T03:00:38.316 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672982 2024-09-01T03:00:38.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-01T03:00:38.316 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:00:38.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-01T03:00:38.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:00:38.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-01T03:00:38.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:00:38.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:00:38.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-01T03:00:38.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:00:38.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-01T03:00:38.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-01T03:00:38.746 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509461 2024-09-01T03:00:38.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:00:39.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-01T03:00:39.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:00:39.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:00:39.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:00:39.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:00:39.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:00:39.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:00:39.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:00:39.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:00:39.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:00:39.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:00:39.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:00:40.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:00:40.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:00:40.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:00:40.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:00:40.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:00:40.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:00:40.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:00:40.656 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-01T03:00:40.656 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-01T03:00:40.656 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-01T03:00:40.656 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-01T03:00:40.656 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-01T03:00:41.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-01T03:00:41.140 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-09-01T03:00:41.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-01T03:00:41.141 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:00:41.538 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:00:41.538 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-01T03:00:09.715177+0000 29'100 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:19:33.434166+0000 0 0 2024-09-01T03:00:41.538 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:00:41.538 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:00:41.538 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:00:41.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:00:41.549 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-01T03:00:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-01T03:00:41.627 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:00:41.627 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T03:00:41.627 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:00:41.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-01T03:00:41.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-01T03:00:41.713 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:00:41.713 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T03:00:41.713 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:00:41.720 INFO:tasks.workunit.client.0.smithi079.stdout:create new divergent objects 2024-09-01T03:00:41.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-01T03:00:41.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-01T03:00:41.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:00:41.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:00:41.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-01T03:00:41.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-01T03:00:41.722 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent objects 2024-09-01T03:00:41.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-01T03:00:41.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-01T03:00:41.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-01T03:00:41.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-01T03:00:41.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-01T03:00:41.724 INFO:tasks.workunit.client.0.smithi079.stdout:remove divergent objects 2024-09-01T03:00:41.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-01T03:00:41.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-01T03:00:41.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-01T03:00:41.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-01T03:00:41.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-01T03:00:41.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-01T03:00:41.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-01T03:00:41.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-01T03:00:41.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-01T03:00:41.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-01T03:00:41.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-01T03:00:41.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-01T03:00:51.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-01T03:00:51.734 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-09-01T03:00:51.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-01T03:00:51.735 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-09-01T03:00:51.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:00:52.112 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:00:52.113 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 active+clean 2024-09-01T03:00:09.715177+0000 29'106 29:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:19:33.434166+0000 0 0 2024-09-01T03:00:52.113 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:00:52.113 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:00:52.113 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:00:52.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-01T03:00:52.125 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:00:52.125 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:00:52.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:00:52.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:00:52.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:00:52.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:00:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-01T03:00:52.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-01T03:00:52.721 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:00:52.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-01T03:00:52.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-01T03:00:53.269 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T03:00:53.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-01T03:00:53.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-01T03:00:53.780 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T03:00:53.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:00:53.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-01T03:00:54.298 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-09-01T03:00:54.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:00:54.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-01T03:00:54.807 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-09-01T03:00:54.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-01T03:00:54.819 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-09-01T03:00:54.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:00:55.193 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:00:55.193 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 stale+active+clean 2024-09-01T03:00:09.715177+0000 29'106 29:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:19:33.434166+0000 0 0 2024-09-01T03:00:55.193 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:00:55.193 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:00:55.193 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:00:55.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:00:55.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-01T03:00:55.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:00:55.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:00:55.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:00:55.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-01T03:00:55.211 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:00:55.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:00:55.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:00:55.213 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-01T03:00:55.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:00:55.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:00:55.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:00:55.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:00:55.242 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:00:55.240+0000 7f4c8a019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:00:55.243 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:00:55.242+0000 7f4c8a019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:00:55.245 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:00:55.243+0000 7f4c8a019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:00:55.639 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:00:55.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:55.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:00:56.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:00:56.335 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:00:56.333+0000 7f4c8a019640 -1 Falling back to public interface 2024-09-01T03:00:57.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:00:57.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:57.066 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:00:57.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:00:57.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:57.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:00:57.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:00:58.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:00:58.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:58.509 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:00:58.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:00:58.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:58.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:00:58.904 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:00:59.852 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:00:59.850+0000 7f4c8a019640 -1 osd.0 29 log_to_monitors true 2024-09-01T03:00:59.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:00:59.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:00:59.907 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:00:59.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:00:59.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:00:59.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:01:00.308 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3573369352,v1:127.0.0.1:6803/3573369352] [v2:127.0.0.1:6804/3573369352,v1:127.0.0.1:6805/3573369352] exists,up b494a832-c285-435b-9e09-53bb38fafb41 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:01:00.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:01:00.313 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:01:00.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:01:00.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:01:00.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-01T03:01:00.315 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:01:00.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:01:00.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:01:00.316 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-01T03:01:00.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:01:00.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:01:00.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:01:00.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:01:00.346 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:01:00.344+0000 7f3f38fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:01:00.353 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:01:00.352+0000 7f3f38fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:01:00.354 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:01:00.353+0000 7f3f38fcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:01:00.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:01:00.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:01:00.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:01:00.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:01:00.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:01:00.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:01:00.795 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:01:00.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:01:00.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:01:00.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:01:01.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:01:01.704 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:01:01.703+0000 7f3f38fcc640 -1 Falling back to public interface 2024-09-01T03:01:02.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:01:02.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:01:02.185 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:01:02.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:01:02.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:01:02.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:01:02.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:01:03.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:01:03.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:01:03.622 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:01:03.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:01:03.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:01:03.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:01:04.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:01:04.592 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:01:04.589+0000 7f3f38fcc640 -1 osd.2 29 log_to_monitors true 2024-09-01T03:01:05.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:01:05.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:01:05.018 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:01:05.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:01:05.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:01:05.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:01:05.417 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/350832333,v1:127.0.0.1:6811/350832333] [v2:127.0.0.1:6812/350832333,v1:127.0.0.1:6813/350832333] exists,up ecd3a08c-2f13-4e17-9be6-31763cf0d625 2024-09-01T03:01:05.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:01:05.417 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:01:05.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:01:05.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:01:05.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-01T03:01:05.949 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-09-01T03:01:05.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:01:05.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-01T03:01:06.526 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-09-01T03:01:06.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:01:06.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-01T03:01:06.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:01:06.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:01:06.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:01:06.539 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:01:06.539 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:01:06.540 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:01:06.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:01:06.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:01:06.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:01:06.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:01:06.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:01:06.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:01:06.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:01:06.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:01:06.593 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:01:07.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:01:07.044 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:01:07.044 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:01:07.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:01:07.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:01:07.044 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:01:07.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-01T03:01:07.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-01T03:01:07.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-01T03:01:07.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:01:07.165 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:01:07.252 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T03:01:07.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T03:01:07.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T03:01:07.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T03:01:07.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:01:07.255 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:01:07.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-01T03:01:07.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-01T03:01:07.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-01T03:01:07.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:01:07.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-01T03:01:07.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:01:07.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:01:07.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-01T03:01:07.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:01:07.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-01T03:01:07.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-01T03:01:07.378 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 154618822660 2024-09-01T03:01:07.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:01:07.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-01T03:01:07.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:01:08.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:01:08.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:01:09.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-01T03:01:09.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:01:09.207 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-01T03:01:09.207 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:01:09.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:01:09.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-01T03:01:09.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:01:09.209 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 163208757251 2024-09-01T03:01:09.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-01T03:01:09.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-01T03:01:09.210 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:01:09.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-01T03:01:09.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:01:09.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:09.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:10.270 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:10.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:10.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:10.678 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:10.678 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:10.678 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:11.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:11.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T03:01:11.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T03:01:11.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T03:01:11.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:11.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T03:01:11.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:11.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:11.310 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:11.310 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:11.310 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:11.310 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:11.311 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:11.311 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:11.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:11.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:11.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:11.691 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:12.216 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:12.217 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:12.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:12.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:12.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T03:01:12.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:12.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T03:01:12.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:12.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:12.964 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:12.964 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:12.964 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:12.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:12.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:12.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:13.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:13.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:13.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:13.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:13.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:13.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:13.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:13.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:13.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:13.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:13.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:13.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:13.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:14.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:14.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:14.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T03:01:14.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:14.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T03:01:14.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:14.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:14.829 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:14.829 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:14.829 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:14.830 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:14.830 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:14.830 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:15.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:15.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:15.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:15.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:15.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:15.744 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:16.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:16.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:16.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T03:01:16.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:16.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T03:01:17.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:17.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:17.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:17.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:17.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:17.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:17.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:17.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:17.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:17.463 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:17.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:17.464 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:17.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:17.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:17.993 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:18.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:18.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:18.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T03:01:18.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:18.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T03:01:20.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:20.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:20.146 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:20.146 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:20.146 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:20.147 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:20.147 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:20.147 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:20.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:20.536 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:20.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:20.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:21.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:21.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:21.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:21.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:21.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:21.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:21.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:21.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:21.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:21.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:21.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:21.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T03:01:21.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:21.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T03:01:24.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:24.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:24.839 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:24.839 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:24.840 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:24.840 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:24.840 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:24.840 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:25.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:25.234 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:25.235 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:25.235 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:25.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:25.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:25.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:25.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:25.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:25.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:25.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:25.761 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:25.761 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:26.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:26.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:26.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T03:01:26.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:26.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T03:01:32.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:32.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:32.730 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:32.730 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:32.730 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:32.730 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:32.730 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:32.730 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:33.123 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:33.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:33.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:33.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:33.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:34.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:34.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:34.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T03:01:34.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:34.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T03:01:41.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:41.525 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:41.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:41.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:41.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:41.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:42.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:42.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:42.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:42.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:42.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:42.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:42.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:42.443 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:42.443 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:42.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:42.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:42.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T03:01:42.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T03:01:43.493 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:01:43.493 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "bc957a62-601e-45d0-ba73-eafd41b52e05", 2024-09-01T03:01:43.493 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T03:01:43.491620+0000", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.494 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T03:01:43.495 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:01:43.496 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T03:01:43.497 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T03:01:43.498 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T02:58:17.355600Z", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T02:58:17.355600Z", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T03:01:43.499 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.500 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.501 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.502 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 41, 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T02:58:18.256759+0000", 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:01:06.606124+0000", 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T03:01:04.796181+0000", 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T03:01:06.277665+0000", 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T03:01:43.503 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T03:01:43.504 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T03:01:43.505 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T03:01:43.505 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T03:01:43.505 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T03:01:43.505 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T03:00:08.447179+0000", 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-09-01T03:01:43.506 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T03:01:43.507 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T03:01:43.508 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "29", 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T03:01:43.509 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T03:01:43.510 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T03:01:43.511 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.9900000095367432, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.9900000095367432, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 2, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 2, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-09-01T03:01:43.512 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "b494a832-c285-435b-9e09-53bb38fafb41", 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:01:43.513 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 36, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 40, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 30, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.514 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T03:01:43.515 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.516 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.517 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/3573369352", 2024-09-01T03:01:43.518 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/3573369352", 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3573369352", 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3573369352", 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.519 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "690f6dfa-5c78-4412-94fc-a2759be4428d", 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 23, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:01:43.520 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.521 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.522 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.523 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:01:43.524 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/920365757", 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/920365757", 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/920365757", 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/920365757", 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.525 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "ecd3a08c-2f13-4e17-9be6-31763cf0d625", 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 38, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:01:43.526 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.527 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.528 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.529 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.530 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/350832333", 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/350832333", 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/350832333", 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/350832333", 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T03:01:43.531 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:00:52.557435+0000", 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:58:52.896813+0000", 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:00:53.066721+0000", 2024-09-01T03:01:43.532 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:59:28.661319+0000", 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:00:53.608375+0000", 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:01:43.533 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:00:04.573642+0000", 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T03:01:43.534 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T03:01:43.535 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/3573369352,v1:127.0.0.1:6805/3573369352]", 2024-09-01T03:01:43.536 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:01:43.537 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:58:24.724491Z", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:01:43.538 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/3573369352,v1:127.0.0.1:6803/3573369352]", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3573369352,v1:127.0.0.1:6809/3573369352]", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3573369352,v1:127.0.0.1:6807/3573369352]", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:01:43.539 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.540 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/920365757,v1:127.0.0.1:6813/920365757]", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:01:43.541 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:59:00.099009Z", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:01:43.542 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/920365757,v1:127.0.0.1:6811/920365757]", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/920365757,v1:127.0.0.1:6817/920365757]", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/920365757,v1:127.0.0.1:6815/920365757]", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:01:43.543 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:01:43.544 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/350832333,v1:127.0.0.1:6813/350832333]", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:01:43.545 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:59:36.258476Z", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:01:43.546 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/350832333,v1:127.0.0.1:6811/350832333]", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/350832333,v1:127.0.0.1:6817/350832333]", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/350832333,v1:127.0.0.1:6815/350832333]", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:01:43.547 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T03:01:43.548 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 36 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 38 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.549 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 41, 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:01:43.550 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.551 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T03:01:43.552 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T03:01:43.553 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T03:01:43.554 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T03:01:43.555 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.556 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:01:43.557 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:01:43.558 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:01:43.559 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.560 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:01:43.561 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:01:43.562 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T03:01:43.563 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T03:01:43.564 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.565 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T03:01:43.566 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:01:43.566 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T02:58:18:231895+0000", 2024-09-01T03:01:43.566 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T03:01:43.566 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T03:01:43.566 INFO:tasks.workunit.client.0.smithi079.stderr:report 1963749296 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T03:01:43.567 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.568 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T03:01:43.569 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48300, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:01:43.570 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:01:43.571 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:01:43.572 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:01:43.573 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:01:43.574 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2104, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2154496, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 418308, 2024-09-01T03:01:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:01:43.576 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:01:43.577 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2104, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-09-01T03:01:43.578 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2154496, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 418308, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:01:43.579 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:01:43.580 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48300, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:01:43.581 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 100, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 100, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:01:43.582 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:01:43.583 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-09-01T03:01:43.584 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 143276, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 16, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 16, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T03:01:43.585 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 154618822667 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672985 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:01:43.586 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 163208757258 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:01:43.587 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T03:01:43.588 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 4, 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:00:30.963817+0000", 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "services": { 2024-09-01T03:01:43.589 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": { 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "daemons": { 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": "", 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "2": { 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "start_epoch": 0, 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "start_stamp": "0.000000", 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "gid": 0, 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "metadata": {}, 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: "task_status": {} 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.590 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 92, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 378, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 519, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:01:43.591 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:01:43.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T03:01:43.592 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-01T03:01:43.592 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-09-01T03:01:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-01T03:01:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-01T03:01:43.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:01:43.873 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:01:43.873 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 29'100 41:127 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:01:43.873 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:01:43.873 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:01:43.874 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:01:43.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-01T03:01:43.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:01:43.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-01T03:01:43.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:01:43.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:01:43.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:01:43.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:01:43.997 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:01:43.997 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:01:43.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:01:43.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:01:43.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:01:44.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:01:44.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:01:44.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:01:44.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:01:44.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:01:44.051 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:01:44.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:01:44.491 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:01:44.491 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:01:44.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:01:44.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:01:44.492 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:01:44.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822669 2024-09-01T03:01:44.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822669 2024-09-01T03:01:44.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822669' 2024-09-01T03:01:44.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:01:44.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:01:44.699 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T03:01:44.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T03:01:44.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T03:01:44.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T03:01:44.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:01:44.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:01:44.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757261 2024-09-01T03:01:44.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757261 2024-09-01T03:01:44.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822669 2-163208757261' 2024-09-01T03:01:44.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:01:44.821 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822669 2024-09-01T03:01:44.821 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:01:44.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:01:44.823 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822669 2024-09-01T03:01:44.823 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:01:44.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822669 2024-09-01T03:01:44.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2024-09-01T03:01:44.824 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 154618822669 2024-09-01T03:01:44.824 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:01:45.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822669 -lt 154618822669 2024-09-01T03:01:45.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:01:45.226 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757261 2024-09-01T03:01:45.226 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:01:45.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:01:45.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757261 2024-09-01T03:01:45.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:01:45.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757261 2024-09-01T03:01:45.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-09-01T03:01:45.229 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 163208757261 2024-09-01T03:01:45.229 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:01:45.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757261 -lt 163208757261 2024-09-01T03:01:45.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:01:45.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:45.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:46.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:01:46.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:46.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:46.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:46.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:46.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:46.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:46.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:46.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:46.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:46.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:46.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:47.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:47.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T03:01:47.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T03:01:47.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T03:01:47.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:47.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T03:01:47.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:47.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:47.221 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:47.221 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:47.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:47.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:47.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:47.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:47.600 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:47.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:47.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:47.601 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:48.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:48.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:48.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:48.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:48.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:48.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T03:01:48.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:48.694 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T03:01:48.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:48.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:48.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:48.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:48.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:48.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:48.896 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:48.896 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:49.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:49.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:49.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:49.277 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:49.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:49.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:50.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:50.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:50.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T03:01:50.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:50.389 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:50.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:51.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:51.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:51.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:51.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:51.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:51.702 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:52.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:52.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:52.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T03:01:52.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:52.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:53.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:53.067 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:53.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:53.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:53.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:53.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:53.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:53.979 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:53.979 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T03:01:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T03:01:56.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:01:56.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:01:56.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:01:56.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:01:56.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:01:56.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:01:56.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:01:56.103 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:01:56.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:01:56.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:01:56.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:01:56.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:01:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:01:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:01:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:01:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:01:57.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:01:57.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:01:57.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:01:57.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:01:57.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:01:57.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:01:57.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:01:57.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T03:01:57.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:01:57.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T03:02:00.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:02:00.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:02:00.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:02:00.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:02:00.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:02:00.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:02:00.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:02:00.756 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:02:01.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:02:01.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:02:01.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:02:01.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:02:01.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:02:01.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:02:01.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:02:01.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:02:01.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:02:01.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:02:01.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:02:01.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:02:01.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:02:02.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:02:02.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:02:02.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T03:02:02.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:02:02.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T03:02:08.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:02:08.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:02:08.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:02:08.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:02:08.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:02:08.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:02:08.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:02:08.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:02:09.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:02:09.019 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:02:09.019 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:02:09.020 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:02:09.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:02:09.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:02:09.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:02:09.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:02:09.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:02:09.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:02:09.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:02:09.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:02:09.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:02:10.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:02:10.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:02:10.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T03:02:10.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:02:10.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T03:02:17.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:02:17.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:02:17.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:02:17.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:02:17.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:02:17.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:02:17.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:02:17.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:02:17.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:02:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:02:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:02:17.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:02:18.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:02:18.341 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:02:18.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:02:18.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:02:18.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T03:02:18.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T03:02:19.415 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "bc957a62-601e-45d0-ba73-eafd41b52e05", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T03:02:19.413954+0000", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:02:19.416 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:02:19.417 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 2 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.418 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T03:02:19.419 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.420 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.421 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:02:19.421 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.422 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.422 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T03:02:19.422 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.422 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T03:02:19.422 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T02:58:17.355600Z", 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T02:58:17.355600Z", 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T03:02:19.423 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T03:02:19.424 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T03:02:19.425 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T03:02:19.426 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.427 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 41, 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T02:58:18.256759+0000", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:01:06.606124+0000", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T03:01:04.796181+0000", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T03:01:06.277665+0000", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T03:02:19.428 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T03:02:19.429 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T03:00:08.447179+0000", 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T03:02:19.430 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-09-01T03:02:19.431 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "29", 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T03:02:19.432 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T03:02:19.433 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T03:02:19.434 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T03:02:19.434 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T03:02:19.434 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T03:02:19.434 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T03:02:19.434 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T03:02:19.434 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-09-01T03:02:19.435 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T03:02:19.435 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T03:02:19.435 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.435 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T03:02:19.435 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T03:02:19.435 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-09-01T03:02:19.436 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 2.9900000095367432, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 2.9900000095367432, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 2, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 2, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.437 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "b494a832-c285-435b-9e09-53bb38fafb41", 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 36, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 40, 2024-09-01T03:02:19.438 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 30, 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.439 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.440 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.440 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T03:02:19.440 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.440 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.440 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.440 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.441 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:02:19.441 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.441 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.441 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.441 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T03:02:19.441 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.442 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T03:02:19.443 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.444 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.444 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.444 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.444 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T03:02:19.444 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.444 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.445 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.445 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.445 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:02:19.445 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.445 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.445 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.446 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T03:02:19.446 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.446 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.446 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.446 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.446 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T03:02:19.447 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3573369352 2024-09-01T03:02:19.447 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.447 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.447 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.447 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/3573369352", 2024-09-01T03:02:19.447 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/3573369352", 2024-09-01T03:02:19.448 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3573369352", 2024-09-01T03:02:19.448 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3573369352", 2024-09-01T03:02:19.448 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:02:19.448 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:02:19.448 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:02:19.448 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.449 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.449 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.449 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:02:19.449 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "690f6dfa-5c78-4412-94fc-a2759be4428d", 2024-09-01T03:02:19.449 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T03:02:19.449 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:02:19.450 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:02:19.450 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:02:19.450 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T03:02:19.450 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T03:02:19.450 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T03:02:19.450 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 23, 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.451 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:02:19.452 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.452 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.452 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.452 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.452 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:02:19.452 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.453 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.453 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.453 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.453 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:02:19.453 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.454 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.455 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:02:19.455 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.455 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.455 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.455 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.455 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:02:19.456 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.456 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.456 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.456 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:02:19.456 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.456 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.457 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.457 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.457 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:02:19.457 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.457 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.457 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.458 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.458 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:02:19.458 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.458 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.458 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:02:19.459 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 920365757 2024-09-01T03:02:19.460 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.460 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.460 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.460 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/920365757", 2024-09-01T03:02:19.460 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/920365757", 2024-09-01T03:02:19.461 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/920365757", 2024-09-01T03:02:19.461 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/920365757", 2024-09-01T03:02:19.461 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:02:19.461 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T03:02:19.461 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.461 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.462 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.462 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:02:19.462 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "ecd3a08c-2f13-4e17-9be6-31763cf0d625", 2024-09-01T03:02:19.462 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:02:19.462 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:02:19.462 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:02:19.463 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:02:19.463 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-09-01T03:02:19.463 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T03:02:19.463 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 38, 2024-09-01T03:02:19.463 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-09-01T03:02:19.463 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:02:19.464 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.465 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:02:19.466 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.467 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:02:19.468 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 350832333 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/350832333", 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/350832333", 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/350832333", 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/350832333", 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:02:19.469 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:00:52.557435+0000", 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:58:52.896813+0000", 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:02:19.470 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:00:53.066721+0000", 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T02:59:28.661319+0000", 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:02:19.471 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:00:53.608375+0000", 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:00:04.573642+0000", 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T03:02:19.472 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T03:02:19.473 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:02:19.474 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/3573369352,v1:127.0.0.1:6805/3573369352]", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T03:02:19.475 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:58:24.724491Z", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:02:19.476 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/3573369352,v1:127.0.0.1:6803/3573369352]", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3573369352,v1:127.0.0.1:6809/3573369352]", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3573369352,v1:127.0.0.1:6807/3573369352]", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:02:19.477 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/920365757,v1:127.0.0.1:6813/920365757]", 2024-09-01T03:02:19.478 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:02:19.479 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:59:00.099009Z", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:02:19.480 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/920365757,v1:127.0.0.1:6811/920365757]", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/920365757,v1:127.0.0.1:6817/920365757]", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/920365757,v1:127.0.0.1:6815/920365757]", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:02:19.481 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/350832333,v1:127.0.0.1:6813/350832333]", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:02:19.482 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:02:19.483 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T02:59:36.258476Z", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:02:19.484 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/350832333,v1:127.0.0.1:6811/350832333]", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/350832333,v1:127.0.0.1:6817/350832333]", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/350832333,v1:127.0.0.1:6815/350832333]", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:02:19.485 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T03:02:19.486 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 36 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 38 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.488 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 41, 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T03:02:19.489 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.490 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T03:02:19.491 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T03:02:19.492 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.493 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:02:19.494 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:02:19.495 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:02:19.496 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:02:19.497 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:02:19.498 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:02:19.499 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.500 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.501 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T03:02:19.502 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T03:02:19.503 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T02:58:18:231895+0000", 2024-09-01T03:02:19.504 INFO:tasks.workunit.client.0.smithi079.stderr:report 263662239 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T03:02:19.505 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T03:02:19.506 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-09-01T03:02:19.507 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 47971, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:02:19.508 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:02:19.509 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:02:19.510 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:02:19.511 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 17, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 17, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T03:02:19.512 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2104, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-09-01T03:02:19.513 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2154496, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 417650, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:02:19.514 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:02:19.515 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83660, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2104, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T03:02:19.516 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489140, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036879360, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2154496, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 417650, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:02:19.517 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:02:19.518 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 47971, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 100, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 300, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:02:19.519 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 100, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 100, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:02:19.520 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:02:19.521 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 142618, 2024-09-01T03:02:19.522 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 17, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 17, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 2, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 2, 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T03:02:19.523 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 154618822676 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672985 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 163208757267 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.524 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.525 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:02:19.526 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 6, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:02:04.980542+0000", 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 110, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 442, 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T03:02:19.527 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 601, 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T03:02:19.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-01T03:02:19.529 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-09-01T03:02:19.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:19.792 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:19.792 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 41'101 41:128 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:02:19.792 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:19.793 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:19.793 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:19.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:02:19.804 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-01T03:02:19.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok set_recovery_delay 100000 2024-09-01T03:02:19.885 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:02:19.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-01T03:02:19.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-01T03:02:19.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T03:02:19.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T03:02:19.894 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:02:19.894 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:02:19.894 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:02:19.895 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T03:02:19.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-01T03:02:19.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok set_recovery_delay 100000 2024-09-01T03:02:19.974 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:02:19.981 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-09-01T03:02:19.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-01T03:02:19.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:20.352 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:20.352 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 41'101 41:128 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:02:20.352 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:20.352 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:20.352 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:20.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-01T03:02:20.989 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-09-01T03:02:21.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:02:21.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:02:21.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:02:21.005 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:02:21.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:02:21.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T03:02:21.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:02:21.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:02:21.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:02:21.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:02:21.038 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:21.037+0000 7fbe8843b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:02:21.082 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:21.080+0000 7fbe8843b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:02:21.083 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:21.081+0000 7fbe8843b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:02:21.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-01T03:02:21.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-01T03:02:21.927 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:21.926+0000 7fbe8843b640 -1 Falling back to public interface 2024-09-01T03:02:24.854 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:24.852+0000 7fbe8843b640 -1 osd.1 29 log_to_monitors true 2024-09-01T03:02:26.398 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-09-01T03:02:26.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-01T03:02:26.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:26.779 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:26.780 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 41'101 41:128 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:02:26.780 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:26.780 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:26.780 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-01T03:02:26.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok set_recovery_delay 100000 2024-09-01T03:02:26.872 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:02:26.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-01T03:02:27.766 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-09-01T03:02:27.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-01T03:02:27.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:02:27.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:02:27.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:02:27.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:02:27.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:02:27.779 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:02:27.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:02:27.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:02:27.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:02:28.176 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3573369352,v1:127.0.0.1:6803/3573369352] [v2:127.0.0.1:6804/3573369352,v1:127.0.0.1:6805/3573369352] exists,up b494a832-c285-435b-9e09-53bb38fafb41 2024-09-01T03:02:28.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:02:28.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:02:28.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:02:28.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:02:28.178 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:02:28.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:02:28.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:02:28.617 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2090982576,v1:127.0.0.1:6819/2090982576] [v2:127.0.0.1:6820/2090982576,v1:127.0.0.1:6821/2090982576] exists,up 690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T03:02:28.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:02:28.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:02:28.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:02:28.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:02:28.619 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:02:28.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:02:28.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:02:29.013 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/350832333,v1:127.0.0.1:6811/350832333] [v2:127.0.0.1:6812/350832333,v1:127.0.0.1:6813/350832333] exists,up ecd3a08c-2f13-4e17-9be6-31763cf0d625 2024-09-01T03:02:29.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:02:29.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:02:29.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:02:29.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:29.383 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:29.384 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 41'101 43:132 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:02:29.384 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:29.384 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:29.384 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:29.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-01T03:02:29.394 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-09-01T03:02:29.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:29.761 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:29.761 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 41'101 43:132 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:02:29.761 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:29.762 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:29.762 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:29.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-01T03:02:29.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-01T03:02:29.892 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-09-01T03:02:29.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:30.260 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:30.261 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-01T03:01:06.831039+0000 41'101 43:132 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:54:24.374675+0000 0 0 2024-09-01T03:02:30.261 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:30.261 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:30.261 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:30.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-01T03:02:30.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:02:30.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:02:30.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:02:30.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:02:30.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:02:30.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:02:30.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.22149.out 2024-09-01T03:02:30.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.22149.out 2024-09-01T03:02:30.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:02:30.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:02:30.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-01T03:02:30.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:02:30.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-01T03:02:30.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.22149.out 2024-09-01T03:02:31.813 INFO:tasks.workunit.client.0.smithi079.stderr:Exporting 1.0 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=46/48 n=101 ec=23/23 lis/c=46/23 les/c/f=48/24/0 sis=46) 2024-09-01T03:02:31.815 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-01T03:02:31.815 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-01T03:02:31.815 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-01T03:02:31.816 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-01T03:02:31.816 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-01T03:02:31.816 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-01T03:02:31.817 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-01T03:02:31.817 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:169157fa:::existing_81:head# 2024-09-01T03:02:31.817 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:17731382:::existing_58:head# 2024-09-01T03:02:31.818 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-01T03:02:31.818 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-01T03:02:31.818 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-01T03:02:31.819 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-01T03:02:31.820 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-01T03:02:31.820 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:28566edb:::existing_78:head# 2024-09-01T03:02:31.820 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:2a101182:::existing_20:head# 2024-09-01T03:02:31.820 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-01T03:02:31.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-01T03:02:31.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-01T03:02:31.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-01T03:02:31.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-01T03:02:31.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-01T03:02:31.822 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-01T03:02:31.823 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-01T03:02:31.823 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-01T03:02:31.824 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-01T03:02:31.824 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:53588912:::existing_9:head# 2024-09-01T03:02:31.824 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-01T03:02:31.826 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-01T03:02:31.826 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-01T03:02:31.826 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-01T03:02:31.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:602f83fe:::foo:head# 2024-09-01T03:02:31.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-01T03:02:31.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:628918ca:::existing_79:head# 2024-09-01T03:02:31.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-01T03:02:31.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-01T03:02:31.827 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:750ace21:::existing_19:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-01T03:02:31.829 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:79881152:::existing_63:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:81490677:::existing_76:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-01T03:02:31.831 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:92731e71:::existing_89:head# 2024-09-01T03:02:31.832 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:928205cf:::existing_82:head# 2024-09-01T03:02:31.832 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:93be2267:::existing_24:head# 2024-09-01T03:02:31.833 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-01T03:02:31.833 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-01T03:02:31.833 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-01T03:02:31.834 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-01T03:02:31.834 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-01T03:02:31.834 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a504a533:::existing_90:head# 2024-09-01T03:02:31.834 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a691337e:::existing_61:head# 2024-09-01T03:02:31.835 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-01T03:02:31.835 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-01T03:02:31.835 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-01T03:02:31.835 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-01T03:02:31.835 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-01T03:02:31.836 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:b343e277:::existing_88:head# 2024-09-01T03:02:31.836 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-01T03:02:31.836 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-01T03:02:31.837 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-01T03:02:31.837 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-01T03:02:31.837 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:bebec877:::existing_74:head# 2024-09-01T03:02:31.838 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-01T03:02:31.838 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c156033e:::existing_100:head# 2024-09-01T03:02:31.838 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-01T03:02:31.839 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-01T03:02:31.839 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-01T03:02:31.840 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-01T03:02:31.840 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-01T03:02:31.841 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-01T03:02:31.841 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-01T03:02:31.841 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-01T03:02:31.841 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-01T03:02:31.842 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-01T03:02:31.842 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-01T03:02:31.842 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-01T03:02:31.842 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-01T03:02:31.842 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-01T03:02:31.844 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:e255b710:::existing_55:head# 2024-09-01T03:02:31.844 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-01T03:02:31.844 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-01T03:02:31.844 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-01T03:02:31.844 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f6809456:::existing_11:head# 2024-09-01T03:02:31.844 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-01T03:02:31.845 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-01T03:02:31.845 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-01T03:02:31.845 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-01T03:02:31.845 INFO:tasks.workunit.client.0.smithi079.stderr:Export successful 2024-09-01T03:02:31.846 INFO:tasks.workunit.client.0.smithi079.stdout: marking collection for removal 2024-09-01T03:02:31.846 INFO:tasks.workunit.client.0.smithi079.stdout:setting '_remove' omap key 2024-09-01T03:02:31.846 INFO:tasks.workunit.client.0.smithi079.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-01T03:02:32.047 INFO:tasks.workunit.client.0.smithi079.stderr:Remove successful 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.22149.out 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-01T03:02:32.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.22149.out 2024-09-01T03:02:33.585 INFO:tasks.workunit.client.0.smithi079.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={23=1}} deleted_pools ) 2024-09-01T03:02:33.585 INFO:tasks.workunit.client.0.smithi079.stdout:Importing pgid 1.0 2024-09-01T03:02:33.586 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-01T03:02:33.586 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.667 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-01T03:02:33.667 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.708 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-01T03:02:33.708 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.750 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-01T03:02:33.750 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.792 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-01T03:02:33.792 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.834 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-01T03:02:33.834 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.875 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-01T03:02:33.875 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.917 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:169157fa:::existing_81:head# 2024-09-01T03:02:33.917 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:33.959 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:17731382:::existing_58:head# 2024-09-01T03:02:33.959 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.000 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-01T03:02:34.001 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.034 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-01T03:02:34.034 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.068 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-01T03:02:34.068 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.101 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-01T03:02:34.101 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.134 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-01T03:02:34.135 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.168 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:28566edb:::existing_78:head# 2024-09-01T03:02:34.168 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.201 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:2a101182:::existing_20:head# 2024-09-01T03:02:34.202 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.235 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-01T03:02:34.235 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.269 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-01T03:02:34.269 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.302 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-01T03:02:34.302 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.336 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-01T03:02:34.336 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.369 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-01T03:02:34.369 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.402 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-01T03:02:34.403 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.436 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-01T03:02:34.436 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.470 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-01T03:02:34.470 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.503 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-01T03:02:34.503 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.536 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-01T03:02:34.536 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.570 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:53588912:::existing_9:head# 2024-09-01T03:02:34.570 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.603 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-01T03:02:34.603 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.637 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-01T03:02:34.637 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.670 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-01T03:02:34.670 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.704 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-01T03:02:34.704 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.737 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:602f83fe:::foo:head# 2024-09-01T03:02:34.737 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.770 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-01T03:02:34.770 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.804 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:628918ca:::existing_79:head# 2024-09-01T03:02:34.804 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.837 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-01T03:02:34.838 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.871 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-01T03:02:34.871 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.904 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-01T03:02:34.905 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.938 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-01T03:02:34.938 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:34.971 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-01T03:02:34.972 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.005 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-01T03:02:35.005 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.039 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-01T03:02:35.040 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.106 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-01T03:02:35.107 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.161 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:750ace21:::existing_19:head# 2024-09-01T03:02:35.162 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.223 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-01T03:02:35.224 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.274 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:79881152:::existing_63:head# 2024-09-01T03:02:35.274 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.324 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-01T03:02:35.324 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.359 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:81490677:::existing_76:head# 2024-09-01T03:02:35.359 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.400 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-01T03:02:35.401 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.442 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-01T03:02:35.442 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.484 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-01T03:02:35.484 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.526 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-01T03:02:35.526 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.567 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:92731e71:::existing_89:head# 2024-09-01T03:02:35.568 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.609 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:928205cf:::existing_82:head# 2024-09-01T03:02:35.609 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.651 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:93be2267:::existing_24:head# 2024-09-01T03:02:35.651 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.693 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-01T03:02:35.693 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.726 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-01T03:02:35.726 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.761 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-01T03:02:35.761 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.795 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-01T03:02:35.795 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.828 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-01T03:02:35.828 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.861 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a504a533:::existing_90:head# 2024-09-01T03:02:35.861 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.895 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a691337e:::existing_61:head# 2024-09-01T03:02:35.895 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.928 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-01T03:02:35.928 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.962 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-01T03:02:35.962 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:35.995 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-01T03:02:35.995 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.028 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-01T03:02:36.029 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.079 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-01T03:02:36.079 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.112 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:b343e277:::existing_88:head# 2024-09-01T03:02:36.113 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.146 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-01T03:02:36.146 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.179 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-01T03:02:36.179 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.213 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-01T03:02:36.213 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.246 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-01T03:02:36.246 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.296 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:bebec877:::existing_74:head# 2024-09-01T03:02:36.296 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.330 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-01T03:02:36.330 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.363 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c156033e:::existing_100:head# 2024-09-01T03:02:36.364 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.397 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-01T03:02:36.397 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.430 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-01T03:02:36.430 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.463 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-01T03:02:36.463 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.497 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-01T03:02:36.497 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.547 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-01T03:02:36.547 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.581 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-01T03:02:36.581 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.614 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-01T03:02:36.614 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.648 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-01T03:02:36.648 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.681 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-01T03:02:36.681 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.715 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-01T03:02:36.715 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.748 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-01T03:02:36.748 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.781 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-01T03:02:36.782 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.823 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-01T03:02:36.823 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.857 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-01T03:02:36.857 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.890 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:e255b710:::existing_55:head# 2024-09-01T03:02:36.890 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.924 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-01T03:02:36.924 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.957 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-01T03:02:36.957 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:36.991 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-01T03:02:36.991 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:37.024 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f6809456:::existing_11:head# 2024-09-01T03:02:37.025 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:37.070 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-01T03:02:37.070 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:37.116 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-01T03:02:37.117 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:37.178 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-01T03:02:37.178 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:37.233 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-01T03:02:37.234 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:02:37.300 INFO:tasks.workunit.client.0.smithi079.stdout:write_pg epoch 48 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=46/48 n=101 ec=23/23 lis/c=46/23 les/c/f=48/24/0 sis=46) 2024-09-01T03:02:37.350 INFO:tasks.workunit.client.0.smithi079.stdout:Import successful 2024-09-01T03:02:37.870 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-09-01T03:02:37.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-01T03:02:37.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:02:38.250 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:02:38.250 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 4 4 0 0 48454 0 0 12 90 12 active+recovering+degraded 2024-09-01T03:02:28.503541+0000 48'102 48:132 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 no scrub is scheduled 0 0 2024-09-01T03:02:38.250 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:02:38.250 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:02:38.251 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:02:38.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:02:38.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:02:38.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:02:38.267 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:02:38.268 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:02:38.268 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:02:38.268 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T03:02:38.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:02:38.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:02:38.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:02:38.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:02:38.299 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:38.298+0000 7ff1489c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:02:38.301 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:38.300+0000 7ff1489c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:02:38.302 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:38.301+0000 7ff1489c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:02:38.690 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:02:38.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:02:38.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:02:39.148 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2090982576,v1:127.0.0.1:6819/2090982576] [v2:127.0.0.1:6820/2090982576,v1:127.0.0.1:6821/2090982576] exists,up 690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:02:39.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:02:39.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:02:39.150 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:02:39.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:02:39.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:02:39.395 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:39.394+0000 7ff1489c1640 -1 Falling back to public interface 2024-09-01T03:02:39.608 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2090982576,v1:127.0.0.1:6819/2090982576] [v2:127.0.0.1:6820/2090982576,v1:127.0.0.1:6821/2090982576] exists,up 690f6dfa-5c78-4412-94fc-a2759be4428d 2024-09-01T03:02:39.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:02:39.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:02:39.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:02:39.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-01T03:02:42.325 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:02:42.323+0000 7ff1489c1640 -1 osd.1 48 log_to_monitors true 2024-09-01T03:02:59.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-01T03:02:59.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T03:02:59.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T03:02:59.611 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:02:59.611 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:02:59.612 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:02:59.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T03:02:59.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-01T03:02:59.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok dump_ops_in_flight 2024-09-01T03:02:59.699 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:02:59.699 INFO:tasks.workunit.client.0.smithi079.stdout: "ops": [], 2024-09-01T03:02:59.699 INFO:tasks.workunit.client.0.smithi079.stdout: "num_ops": 0 2024-09-01T03:02:59.700 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:02:59.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-01T03:02:59.707 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-09-01T03:02:59.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:03:00.081 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:03:00.081 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-01T03:02:43.958077+0000 48'102 52:154 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T04:03:28.833748+0000 0 0 2024-09-01T03:03:00.081 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:03:00.081 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:03:00.081 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:03:00.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-01T03:03:00.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-01T03:03:00.203 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:03:00.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-01T03:03:00.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-01T03:03:00.323 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:03:00.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-01T03:03:00.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-01T03:03:00.444 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:03:00.453 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-09-01T03:03:00.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-01T03:03:00.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-01T03:03:00.824 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:03:00.824 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-01T03:02:43.958077+0000 48'102 52:154 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:00:08.510043+0000 0'0 2024-09-01T03:00:08.510043+0000 0 0 periodic scrub scheduled @ 2024-09-02T04:03:28.833748+0000 0 0 2024-09-01T03:03:00.824 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:03:00.824 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:03:00.824 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:03:00.835 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-01T03:03:00.837 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-01T03:03:00.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:03:00.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-01T03:03:00.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:03:00.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-01T03:03:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:03:00.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-01T03:03:00.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:03:00.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-01T03:03:00.979 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-01T03:03:00.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:03:00.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-01T03:03:01.013 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-01T03:03:01.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:03:01.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-01T03:03:01.076 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-01T03:03:01.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-01T03:03:01.078 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-01T03:03:01.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:03:01.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.441+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4641.0:1 2024-09-01T03:00:41.782398+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:03:01.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4641.0:1 2024-09-01T03:00:41.782398+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:03:01.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:03:01.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-01T03:03:01.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4760.0:1 2024-09-01T03:01:43.917018+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false, already merged 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4630.0:1 2024-09-01T03:00:41.758214+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4630.0:1 2024-09-01T03:00:41.758214+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4643.0:1 2024-09-01T03:00:41.783303+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4643.0:1 2024-09-01T03:00:41.783303+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-01T03:03:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4643.0:1 2024-09-01T03:00:41.783303+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-01T03:00:41.759339+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-01T03:00:41.759339+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:03:01.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-01T03:00:41.759339+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4642.0:1 2024-09-01T03:00:41.784288+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:03:01.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: keeping 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4642.0:1 2024-09-01T03:00:41.784288+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4642.0:1 2024-09-01T03:00:41.784288+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-01T03:03:01.088 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4634.0:1 2024-09-01T03:00:41.773479+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4634.0:1 2024-09-01T03:00:41.773479+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:02:28.442+0000 7fbe6c3ad640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-01T03:03:01.089 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-01T03:03:01.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:03:01.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-01T03:03:01.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:03:01.096 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-09-01T03:03:01.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-01T03:03:01.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.22149.out 2024-09-01T03:03:01.098 INFO:tasks.workunit.client.0.smithi079.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.22149.out': No such file or directory 2024-09-01T03:03:01.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-01T03:03:01.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-01T03:03:01.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-01T03:03:01.619 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-09-01T03:03:01.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-01T03:03:01.631 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:03:01.631 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:03:01.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:03:01.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:03:01.631 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:03:06.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:03:06.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-01T03:03:06.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:03:06.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:03:06.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:03:06.952 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:03:06.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:03:06.953 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:03:06.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:03:06.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:03:06.955 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:03:06.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:03:06.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:03:06.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:03:06.957 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:03:06.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:03:06.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:03:06.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T03:03:06.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:03:06.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:06.983 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:06.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-01T03:03:06.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:03:06.987 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:03:06.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:03:06.988 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:03:06.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:03:06.988 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:03:06.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:03:06.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:03:06.989 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:03:06.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:03:06.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:03:06.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:03:06.991 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:03:06.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:03:06.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:03:06.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T03:03:06.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:03:06.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:06.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:06.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T03:03:06.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:03:06.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:03:06.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-01T03:03:06.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:03:06.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:06.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:06.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.22149 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-01T03:03:06.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-01T03:03:07.000 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-01T03:03:07.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-01T03:03:07.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-01T03:03:07.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:03:07.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:03:07.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:03:07.434 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:03:07.434 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:07.435 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:07.435 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:03:07.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:03:07.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:03:07.619 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:03:07.619 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:03:07.619 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:03:07.619 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:03:07.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:03:07.620 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:03:07.621 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get fsid 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:03:07.708 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:07.709 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:07.709 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T03:03:07.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:03:07.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get mon_host 2024-09-01T03:03:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-01T03:03:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-01T03:03:07.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:03:07.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:03:07.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:03:07.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-01T03:03:07.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:03:08.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:03:08.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:03:08.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:03:08.639 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:03:08.639 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:08.639 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:08.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:03:08.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:03:08.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:03:08.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-01T03:03:08.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:03:08.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:08.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:03:08.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:03:08.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-01T03:03:08.679 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:03:08.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0bb18a82-93ad-4ae9-8f56-88086e781f9b 2024-09-01T03:03:08.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0bb18a82-93ad-4ae9-8f56-88086e781f9b' 2024-09-01T03:03:08.680 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 0bb18a82-93ad-4ae9-8f56-88086e781f9b 2024-09-01T03:03:08.680 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:03:08.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBs2dNmXoykKRAAoeA1wTitFh1jR/82VWmGGA== 2024-09-01T03:03:08.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBs2dNmXoykKRAAoeA1wTitFh1jR/82VWmGGA=="}' 2024-09-01T03:03:08.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0bb18a82-93ad-4ae9-8f56-88086e781f9b -i td/divergent-priors/0/new.json 2024-09-01T03:03:08.976 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:03:08.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-01T03:03:08.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBs2dNmXoykKRAAoeA1wTitFh1jR/82VWmGGA== --osd-uuid 0bb18a82-93ad-4ae9-8f56-88086e781f9b 2024-09-01T03:03:09.024 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:09.021+0000 7f26b874d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:09.044 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:09.042+0000 7f26b874d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:09.045 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:09.043+0000 7f26b874d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:09.046 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:09.043+0000 7f26b874d640 -1 bdev(0x55e3cc7d1c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:03:09.046 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:09.043+0000 7f26b874d640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-01T03:03:14.547 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-01T03:03:14.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:03:14.548 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:03:14.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:03:14.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:03:15.092 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:03:15.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:03:15.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:03:15.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:03:15.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:03:15.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:03:15.128 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:15.126+0000 7f4360264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:15.131 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:15.129+0000 7f4360264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:15.132 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:15.131+0000 7f4360264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:15.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:03:15.546 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:03:15.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:15.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:15.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:16.228 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:16.227+0000 7f4360264640 -1 Falling back to public interface 2024-09-01T03:03:16.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:16.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:16.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:03:16.964 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:03:16.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:16.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:17.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:18.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:18.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:18.398 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:03:18.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:03:18.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:18.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:18.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:19.434 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:19.432+0000 7f4360264640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:03:19.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:19.840 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:19.841 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:03:19.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:03:19.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:19.841 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:20.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:21.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:21.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:21.351 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:03:21.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:03:21.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:21.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:21.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:22.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:22.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:22.776 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:03:22.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:03:22.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:22.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:23.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:24.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:24.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:24.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:03:24.224 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:03:24.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:24.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:24.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:25.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:25.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:25.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:03:25.653 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:03:25.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:25.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:26.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:27.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:27.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:27.186 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:03:27.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:03:27.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:27.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:27.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:28.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:28.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:28.622 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:03:28.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:03:28.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:28.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:29.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:30.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:30.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:30.172 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:03:30.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:03:30.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:30.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:30.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:31.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:31.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:31.592 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:03:31.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:03:31.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:31.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:32.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:33.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:33.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:33.384 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:03:33.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:03:33.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:33.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:34.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:35.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:35.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:35.161 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:03:35.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:03:35.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:35.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:36.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:37.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:37.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:37.202 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:03:37.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:03:37.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:37.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:37.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:38.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:38.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:38.791 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:03:38.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:03:38.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:38.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:39.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:40.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:40.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:40.510 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:03:40.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:03:40.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:40.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:40.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:41.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:41.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:41.966 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:03:41.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:03:41.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:41.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:03:42.417 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1865109012,v1:127.0.0.1:6803/1865109012] [v2:127.0.0.1:6804/1865109012,v1:127.0.0.1:6805/1865109012] exists,up 0bb18a82-93ad-4ae9-8f56-88086e781f9b 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:03:42.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:03:42.424 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:03:42.425 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 427ee48c-66f3-493b-b6c3-1c333687da39' 2024-09-01T03:03:42.426 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:03:42.427 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:03:42.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCO2dNmFfZvGhAAGruhpLpOdw/SFaIa4hm6XA== 2024-09-01T03:03:42.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCO2dNmFfZvGhAAGruhpLpOdw/SFaIa4hm6XA=="}' 2024-09-01T03:03:42.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 427ee48c-66f3-493b-b6c3-1c333687da39 -i td/divergent-priors/1/new.json 2024-09-01T03:03:42.966 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:03:42.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-01T03:03:42.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCO2dNmFfZvGhAAGruhpLpOdw/SFaIa4hm6XA== --osd-uuid 427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:03:43.015 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:43.013+0000 7fd72181d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:43.026 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:43.025+0000 7fd72181d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:43.027 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:43.026+0000 7fd72181d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:43.028 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:43.026+0000 7fd72181d640 -1 bdev(0x562b9c14bc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-01T03:03:43.028 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:43.026+0000 7fd72181d640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-01T03:03:48.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-01T03:03:48.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:03:48.836 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T03:03:48.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T03:03:48.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:03:49.378 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:03:49.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T03:03:49.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:03:49.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:03:49.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:03:49.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:03:49.413 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:49.411+0000 7f5bf6216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:49.421 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:49.418+0000 7f5bf6216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:49.419+0000 7f5bf6216640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:03:49.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T03:03:49.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:03:49.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:03:49.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:03:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:03:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:49.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:03:49.797 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:03:49.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:49.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:50.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:50.514 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:50.512+0000 7f5bf6216640 -1 Falling back to public interface 2024-09-01T03:03:51.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:51.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:51.183 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:03:51.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:03:51.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:51.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:51.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:52.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:52.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:52.613 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:03:52.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:03:52.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:52.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:53.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:53.425 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:03:53.423+0000 7f5bf6216640 -1 osd.1 0 log_to_monitors true 2024-09-01T03:03:54.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:54.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:54.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:03:54.078 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:03:54.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:54.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:55.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:55.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:55.563 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:03:55.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:03:55.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:55.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:55.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:56.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:56.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:56.974 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:03:56.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:03:56.974 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:56.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:57.527 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:03:58.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:03:58.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:03:58.529 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:03:58.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:03:58.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:03:58.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:03:59.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:00.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:00.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:00.023 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:04:00.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:04:00.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:00.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:00.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:01.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:01.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:01.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:04:01.454 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:04:01.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:01.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:01.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:02.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:02.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:02.988 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:04:02.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:04:02.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:02.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:03.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:04.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:04.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:04.463 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:04:04.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:04:04.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:04.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:04.834 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:05.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:05.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:05.836 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:04:05.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:04:05.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:05.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:06.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:07.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:07.383 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:07.384 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:04:07.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:04:07.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:07.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:08.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:09.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:09.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:09.346 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:04:09.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:04:09.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:09.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:10.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:11.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:11.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:11.056 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:04:11.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:04:11.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:11.056 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:11.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:12.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:12.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:12.933 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:04:12.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:04:12.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:12.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:13.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:14.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:14.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:14.803 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:04:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:04:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:15.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:16.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:16.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:16.710 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:04:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:04:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:16.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:04:17.183 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1570722299,v1:127.0.0.1:6811/1570722299] [v2:127.0.0.1:6812/1570722299,v1:127.0.0.1:6813/1570722299] exists,up 427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:04:17.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:04:17.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:04:17.189 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:04:17.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4fb7a0c5-453b-4a68-91b5-29d07b3434d4 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4fb7a0c5-453b-4a68-91b5-29d07b3434d4' 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 4fb7a0c5-453b-4a68-91b5-29d07b3434d4 2024-09-01T03:04:17.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:04:17.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCx2dNmhPiBDBAA9ukMXoyIM97iA3ybhwz6ig== 2024-09-01T03:04:17.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCx2dNmhPiBDBAA9ukMXoyIM97iA3ybhwz6ig=="}' 2024-09-01T03:04:17.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4fb7a0c5-453b-4a68-91b5-29d07b3434d4 -i td/divergent-priors/2/new.json 2024-09-01T03:04:17.832 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:04:17.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-01T03:04:17.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCx2dNmhPiBDBAA9ukMXoyIM97iA3ybhwz6ig== --osd-uuid 4fb7a0c5-453b-4a68-91b5-29d07b3434d4 2024-09-01T03:04:17.881 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:17.879+0000 7fd03342c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:04:17.883 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:17.881+0000 7fd03342c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:04:17.884 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:17.882+0000 7fd03342c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:04:17.884 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:17.883+0000 7fd03342c640 -1 bdev(0x56549bf49c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-01T03:04:17.884 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:17.883+0000 7fd03342c640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-01T03:04:24.080 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-01T03:04:24.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:04:24.081 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T03:04:24.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T03:04:24.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:04:24.620 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:04:24.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T03:04:24.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:04:24.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:04:24.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:04:24.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:04:24.657 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:24.655+0000 7f088d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:04:24.659 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:24.657+0000 7f088d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:04:24.659 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:24.658+0000 7f088d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:25.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:04:25.051 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:04:25.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:25.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:25.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:26.013 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:26.012+0000 7f088d20d640 -1 Falling back to public interface 2024-09-01T03:04:26.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:26.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:26.480 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:04:26.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:04:26.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:26.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:26.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:27.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:27.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:27.891 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:04:27.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:04:27.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:27.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:28.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:29.004 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:04:29.001+0000 7f088d20d640 -1 osd.2 0 log_to_monitors true 2024-09-01T03:04:29.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:29.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:29.303 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:04:29.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:04:29.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:29.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:29.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:30.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:30.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:30.793 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:04:30.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:04:30.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:30.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:31.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:32.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:32.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:32.249 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:04:32.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:04:32.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:32.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:32.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:33.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:33.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:33.653 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:04:33.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:04:33.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:33.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:34.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:35.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:35.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:35.189 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:04:35.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:04:35.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:35.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:35.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:36.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:36.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:36.615 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:04:36.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:04:36.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:36.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:37.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:38.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:38.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:38.319 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:04:38.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:04:38.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:38.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:38.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:39.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:39.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:39.844 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:04:39.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:04:39.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:39.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:40.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:41.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:41.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:41.266 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:04:41.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:04:41.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:41.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:41.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:42.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:42.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:42.901 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:04:42.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:04:42.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:42.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:43.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:44.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:44.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:44.918 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:04:44.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:04:44.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:44.919 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:45.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:46.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:46.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:46.711 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:04:46.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:04:46.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:46.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:47.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:48.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:48.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:48.520 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:04:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:04:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:48.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:49.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:50.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:50.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:50.192 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:04:50.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:04:50.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:50.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:50.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:04:51.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:04:51.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:04:51.952 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:04:51.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:04:51.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:04:51.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:04:52.371 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3603308373,v1:127.0.0.1:6819/3603308373] [v2:127.0.0.1:6820/3603308373,v1:127.0.0.1:6821/3603308373] exists,up 4fb7a0c5-453b-4a68-91b5-29d07b3434d4 2024-09-01T03:04:52.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:04:52.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:04:52.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:04:52.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-01T03:04:53.012 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-09-01T03:04:53.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-01T03:04:53.682 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-09-01T03:04:53.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-01T03:04:54.494 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-09-01T03:04:54.505 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-01T03:04:54.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-01T03:04:55.214 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-09-01T03:04:55.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T03:04:56.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-01T03:04:56.873 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 size to 3 2024-09-01T03:04:56.884 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-01T03:04:57.609 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 min_size to 2 2024-09-01T03:04:57.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-01T03:04:58.296 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 pg_autoscale_mode to on 2024-09-01T03:04:58.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-01T03:04:58.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-01T03:04:58.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1725159898 2024-09-01T03:04:58.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-01T03:04:58.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-01T03:04:58.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-01T03:04:58.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:04:58.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:04:58.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:04:58.753 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:04:58.753 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:04:58.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:04:58.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:04:58.753 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:04:58.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-01T03:04:58.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-01T03:04:58.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-01T03:04:58.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:04:58.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:04:58.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-01T03:04:58.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-01T03:04:58.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672973' 2024-09-01T03:04:58.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:04:58.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:04:59.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-01T03:04:59.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-01T03:04:59.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672973 2-64424509447' 2024-09-01T03:04:59.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:04:59.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-01T03:04:59.118 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:04:59.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:04:59.119 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-01T03:04:59.119 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:04:59.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-01T03:04:59.120 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836502 2024-09-01T03:04:59.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-01T03:04:59.121 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:04:59.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-01T03:04:59.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:05:00.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:05:00.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:05:00.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-01T03:05:00.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:00.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-01T03:05:00.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:00.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:05:00.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-01T03:05:00.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:00.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-01T03:05:00.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-01T03:05:00.998 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672973 2024-09-01T03:05:00.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:05:01.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-01T03:05:01.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:01.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-01T03:05:01.433 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:01.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:05:01.435 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-01T03:05:01.435 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:01.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-01T03:05:01.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-01T03:05:01.437 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509447 2024-09-01T03:05:01.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:05:01.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-01T03:05:01.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-01T03:05:01.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:05:01.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:05:01.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:05:01.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:05:01.870 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:05:01.870 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:05:01.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:05:01.871 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:05:01.871 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:05:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:05:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:05:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:05:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:05:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:05:01.956 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:05:02.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:05:02.385 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:05:02.385 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:05:02.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:05:02.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:05:02.385 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:05:02.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-01T03:05:02.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-01T03:05:02.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-01T03:05:02.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:05:02.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:05:02.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-01T03:05:02.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-01T03:05:02.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-09-01T03:05:02.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:05:02.627 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:05:02.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-01T03:05:02.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-01T03:05:02.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509450' 2024-09-01T03:05:02.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:02.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-01T03:05:02.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:02.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:05:02.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-01T03:05:02.748 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:02.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-01T03:05:02.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-01T03:05:02.750 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836504 2024-09-01T03:05:02.750 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:05:03.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-01T03:05:03.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:03.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-01T03:05:03.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:03.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:05:03.179 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-01T03:05:03.180 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:03.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-01T03:05:03.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-01T03:05:03.181 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672976 2024-09-01T03:05:03.181 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:05:03.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672976 2024-09-01T03:05:03.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:05:04.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:05:04.631 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:05:05.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-01T03:05:05.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:05.021 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-01T03:05:05.021 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:05.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:05:05.022 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-01T03:05:05.022 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:05.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-01T03:05:05.024 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509450 2024-09-01T03:05:05.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-01T03:05:05.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:05:05.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-01T03:05:05.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:05:05.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:05:05.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:05:05.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:05:05.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:05:05.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:05:05.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:05:05.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:05:05.965 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:05:05.966 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:05:05.966 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:05:06.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:05:06.361 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:05:06.361 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:05:06.361 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:05:06.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:05:06.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:05:06.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:05:06.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-01T03:05:06.894 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-01T03:05:07.261 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:05:07.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-01T03:05:07.273 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-09-01T03:05:07.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-01T03:05:07.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:05:07.638 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:05:07.638 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-01T03:04:56.044197+0000 0'0 31:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:38:24.441037+0000 0 0 2024-09-01T03:05:07.638 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:05:07.638 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:05:07.639 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:05:07.650 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-01T03:05:07.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1725159907 2024-09-01T03:05:07.651 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1725159907 - 1725159898 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=9 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 9 -gt 300 ']' 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-01T03:05:07.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-01T03:05:07.654 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-09-01T03:05:07.655 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-01T03:05:07.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:07.655 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-01T03:05:07.816 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:07.817 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-01T03:05:07.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:07.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-01T03:05:08.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:08.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-01T03:05:08.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:08.253 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-01T03:05:08.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:08.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-01T03:05:08.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:08.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-01T03:05:08.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:08.717 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-01T03:05:08.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:08.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-01T03:05:09.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-01T03:05:09.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-01T03:05:09.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-01T03:05:09.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-01T03:05:09.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-01T03:05:09.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-01T03:05:09.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:09.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-01T03:05:10.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-01T03:05:10.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-01T03:05:10.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-01T03:05:10.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-01T03:05:10.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-01T03:05:10.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-01T03:05:10.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:10.859 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-01T03:05:11.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-01T03:05:11.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.161 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-01T03:05:11.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-01T03:05:11.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-01T03:05:11.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-01T03:05:11.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-01T03:05:11.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-01T03:05:11.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:11.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-01T03:05:12.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:12.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-01T03:05:12.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:12.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-01T03:05:12.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:12.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-01T03:05:12.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:12.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-01T03:05:12.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:12.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-01T03:05:12.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:12.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-01T03:05:13.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:13.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-01T03:05:13.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:13.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-01T03:05:13.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:13.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-01T03:05:13.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:13.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-01T03:05:13.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:13.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-01T03:05:13.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:13.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-01T03:05:14.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-01T03:05:14.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-01T03:05:14.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-01T03:05:14.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-01T03:05:14.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-01T03:05:14.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.749 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-01T03:05:14.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:14.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-01T03:05:16.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:16.167 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-01T03:05:16.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:16.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-01T03:05:16.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:16.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-01T03:05:16.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:16.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-01T03:05:16.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:16.896 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-01T03:05:17.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:17.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-01T03:05:17.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:17.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-01T03:05:17.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:17.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-01T03:05:17.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:17.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-01T03:05:17.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:17.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-01T03:05:17.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:17.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-01T03:05:18.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:18.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-01T03:05:18.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:18.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-01T03:05:18.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:18.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-01T03:05:18.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:18.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-01T03:05:18.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:18.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-01T03:05:18.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:18.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-01T03:05:20.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:20.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-01T03:05:20.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:20.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-01T03:05:20.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:20.653 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-01T03:05:20.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:20.829 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-01T03:05:20.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:20.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-01T03:05:21.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:21.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-01T03:05:21.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:21.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-01T03:05:22.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:22.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-01T03:05:22.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:22.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-01T03:05:22.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:22.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-01T03:05:22.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:22.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-01T03:05:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:23.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-01T03:05:23.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:23.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-01T03:05:24.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:24.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-01T03:05:24.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:24.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-01T03:05:24.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:24.716 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-01T03:05:24.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:24.940 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-01T03:05:25.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:25.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-01T03:05:25.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:25.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-01T03:05:25.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:25.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-01T03:05:26.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:26.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-01T03:05:26.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:26.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-01T03:05:26.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:26.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-01T03:05:26.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:26.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-01T03:05:26.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:26.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-01T03:05:27.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:27.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-01T03:05:27.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:27.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-01T03:05:27.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:27.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-01T03:05:27.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:27.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-01T03:05:28.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:28.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-01T03:05:28.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:28.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-01T03:05:28.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:28.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-01T03:05:28.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-01T03:05:28.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-01T03:05:29.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:05:29.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-01T03:05:29.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:05:29.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:05:29.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:05:29.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:05:29.671 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:05:29.671 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:05:29.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:05:29.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:05:29.671 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:05:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:05:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:05:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:05:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:05:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:05:29.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:05:30.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:05:30.181 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:05:30.181 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:05:30.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:05:30.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:05:30.181 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:05:30.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-01T03:05:30.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-01T03:05:30.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-01T03:05:30.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:05:30.375 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:05:30.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-01T03:05:30.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-01T03:05:30.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-09-01T03:05:30.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:05:30.496 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:05:30.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-01T03:05:30.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-01T03:05:30.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509457' 2024-09-01T03:05:30.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:30.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-01T03:05:30.617 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:30.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:05:30.618 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-01T03:05:30.618 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:30.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-01T03:05:30.619 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836512 2024-09-01T03:05:30.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-01T03:05:30.619 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:05:31.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-01T03:05:31.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:31.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-01T03:05:31.010 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:31.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:05:31.011 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-01T03:05:31.011 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:31.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-01T03:05:31.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-01T03:05:31.013 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672983 2024-09-01T03:05:31.013 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:05:31.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-01T03:05:31.439 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:05:31.440 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-01T03:05:31.440 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:05:31.440 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:05:31.441 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-01T03:05:31.441 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:05:31.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-01T03:05:31.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-01T03:05:31.442 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509457 2024-09-01T03:05:31.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:05:31.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509457 2024-09-01T03:05:31.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:05:32.874 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:05:32.875 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:05:33.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509457 2024-09-01T03:05:33.504 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:05:33.505 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:05:33.505 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:05:34.101 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:05:34.586 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-01T03:05:34.586 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:05:34.587 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:05:34.587 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:05:35.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-01T03:05:35.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:05:35.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:05:35.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-01T03:05:35.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-01T03:05:35.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-01T03:05:35.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-01T03:05:35.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-01T03:05:35.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-01T03:05:35.515 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-09-01T03:05:35.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-01T03:05:35.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:05:35.948 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:05:35.948 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:25.242967+0000 36'51 45:131 [1,2,0] 1 [1,2,0] 1 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:21:09.979994+0000 3 0 2024-09-01T03:05:35.948 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:30.144701+0000 42'51 45:133 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T05:25:20.416806+0000 2 0 2024-09-01T03:05:35.948 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:05:23.045178+0000 38'51 45:120 [2,0,1] 2 [2,0,1] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:59:00.085211+0000 4 0 2024-09-01T03:05:35.948 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:29.156438+0000 41'50 45:122 [1,2,0] 1 [1,2,0] 1 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:51:40.732543+0000 3 0 2024-09-01T03:05:35.948 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:24.605281+0000 34'50 45:127 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:20:06.198031+0000 3 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:05:25.056501+0000 41'51 45:121 [2,0,1] 2 [2,0,1] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:28:11.972603+0000 4 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1449 0 0 16 36 16 active+clean 2024-09-01T03:05:30.091882+0000 42'52 45:128 [1,0,2] 1 [1,0,2] 1 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T11:33:29.186713+0000 3 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 0 0 0 1223 0 0 14 36 14 active+clean 2024-09-01T03:05:24.103549+0000 34'50 45:114 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 0 0 0 2189 0 0 16 36 16 active+clean 2024-09-01T03:05:30.144608+0000 39'52 45:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 2898 0 0 16 36 16 active+clean 2024-09-01T03:05:28.167697+0000 41'52 45:134 [2,0,1] 2 [2,0,1] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:00:33.711300+0000 6 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:05:24.325712+0000 34'50 45:126 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:53:12.990510+0000 2 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:05:24.103891+0000 43'51 45:123 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:37:22.226083+0000 4 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:05:28.519212+0000 38'50 45:115 [0,1,2] 0 [0,1,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:23:25.635225+0000 1 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:23.327391+0000 41'51 45:127 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T05:26:45.015486+0000 2 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 966 0 0 15 36 15 active+clean 2024-09-01T03:05:21.164363+0000 41'51 45:120 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T13:33:45.933225+0000 0 0 2024-09-01T03:05:35.949 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:05:20.369651+0000 32'49 45:125 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:42:56.115620+0000 2 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:16.302016+0000 42'50 45:126 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:11:59.623898+0000 2 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:22.112543+0000 41'51 45:121 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:39:12.132149+0000 1 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:05:31.138546+0000 41'51 45:126 [1,0,2] 1 [1,0,2] 1 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:20:56.368555+0000 4 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 0 0 0 1223 0 0 14 36 14 active+clean 2024-09-01T03:05:24.105314+0000 36'50 45:110 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:27.345972+0000 42'51 45:127 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:30:24.125882+0000 2 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:27.067998+0000 34'50 45:131 [2,0,1] 2 [2,0,1] 2 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:31:42.184965+0000 3 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:05:27.563211+0000 34'50 45:125 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:35:02.808070+0000 1 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1706 0 0 15 36 15 active+clean 2024-09-01T03:05:24.103396+0000 42'51 45:115 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:22.429246+0000 42'51 45:127 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:03:39.940441+0000 1 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:26.070166+0000 36'50 45:120 [2,0,1] 2 [2,0,1] 2 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:34:26.020929+0000 3 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 3 0 0 0 0 1705 0 0 15 36 15 active+clean 2024-09-01T03:05:30.246433+0000 36'51 45:109 [1,0,2] 1 [1,0,2] 1 36'51 2024-09-01T03:05:26.523341+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:05:35.950 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 483 0 0 13 36 13 active+clean 2024-09-01T03:05:28.317547+0000 32'49 45:121 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T05:49:42.437978+0000 1 0 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:21.364078+0000 42'51 45:127 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:11:45.126152+0000 2 0 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:05:24.104861+0000 32'49 45:119 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:10:39.258024+0000 2 0 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:05:25.526519+0000 41'51 45:127 [0,1,2] 0 [0,1,2] 0 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:01:21.904797+0000 4 0 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:05:26.313931+0000 42'51 45:123 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:09:04.121272+0000 4 0 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:05:35.951 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:05:35.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:05:35.960 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-01T03:05:35.960 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-01T03:05:35.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-01T03:05:36.037 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:05:36.037 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T03:05:36.037 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:05:36.044 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:05:36.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-01T03:05:36.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T03:05:36.045 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T03:05:36.046 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:05:36.046 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:05:36.046 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:05:36.046 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T03:05:36.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-01T03:05:36.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-01T03:05:36.121 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:05:36.121 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T03:05:36.121 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:05:36.128 INFO:tasks.workunit.client.0.smithi079.stdout:create new divergent objects 2024-09-01T03:05:36.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-01T03:05:36.128 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-01T03:05:36.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:05:36.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:05:36.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-01T03:05:36.130 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent objects 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-01T03:05:36.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-01T03:05:36.132 INFO:tasks.workunit.client.0.smithi079.stdout:remove divergent objects 2024-09-01T03:05:36.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-01T03:05:36.132 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-01T03:05:36.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-01T03:05:36.134 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-01T03:05:36.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-01T03:05:36.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-01T03:05:36.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-01T03:05:36.135 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-01T03:05:36.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-01T03:05:36.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-01T03:05:36.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-01T03:05:46.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-01T03:05:46.143 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-09-01T03:05:46.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-01T03:05:46.143 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-09-01T03:05:46.143 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:05:46.519 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:25.242967+0000 36'51 53:151 [1,2,0] 1 [1,2,0] 1 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:33:45.606178+0000 3 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:30.144701+0000 42'51 53:153 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:48:21.690397+0000 2 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:05:23.045178+0000 38'51 54:143 [2,0,1] 2 [2,0,1] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:10:39.716002+0000 4 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:39.421487+0000 41'50 53:148 [1,0,2] 1 [1,0,2] 1 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:27:27.891740+0000 3 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:39.108382+0000 34'50 53:118 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:39.108302+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:16:07.213169+0000 3 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 2086 0 0 16 36 16 active+clean 2024-09-01T03:05:36.123976+0000 49'52 53:139 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:36.123925+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:56:06.034548+0000 5 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1449 0 0 16 36 16 active+clean 2024-09-01T03:05:33.760957+0000 42'52 53:154 [1,2,0] 1 [1,2,0] 1 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:11:32.436877+0000 3 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:05:34.180660+0000 34'50 53:142 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T09:37:32.050402+0000 2 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 0 0 0 1932 0 0 16 36 16 active+clean 2024-09-01T03:05:35.137001+0000 39'52 53:146 [1,0,2] 1 [1,0,2] 1 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:14:09.607181+0000 4 0 2024-09-01T03:05:46.520 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 2898 0 0 16 36 16 active+clean 2024-09-01T03:05:28.167697+0000 41'52 54:157 [2,0,1] 2 [2,0,1] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:18:50.253068+0000 6 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:05:24.325712+0000 34'50 53:146 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:49:56.676661+0000 2 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:05:24.103891+0000 43'51 53:143 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:05:19.446048+0000 4 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:05:28.519212+0000 38'50 55:141 [0,1,2] 0 [0,1,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:43:48.006893+0000 1 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:23.327391+0000 41'51 53:147 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:42:47.300526+0000 2 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 966 0 0 15 36 15 active+clean 2024-09-01T03:05:21.164363+0000 41'51 54:143 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:44:09.307422+0000 0 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:05:20.369651+0000 32'49 53:145 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:24:09.248391+0000 2 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:16.302016+0000 42'50 53:146 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:18:25.490803+0000 2 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:22.112543+0000 41'51 54:144 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:23:48.931660+0000 1 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 0 0 0 1449 0 0 17 36 17 active+clean 2024-09-01T03:05:31.138546+0000 49'53 53:148 [1,0,2] 1 [1,0,2] 1 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:56:24.651716+0000 4 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 1932 0 0 14 36 14 active+clean 2024-09-01T03:05:33.203146+0000 36'50 53:138 [1,2,0] 1 [1,2,0] 1 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:42:34.647837+0000 4 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:27.345972+0000 42'51 53:147 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T13:48:26.672497+0000 2 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 2 0 0 0 0 966 0 0 15 36 15 active+clean 2024-09-01T03:05:27.067998+0000 49'51 54:155 [2,0,1] 2 [2,0,1] 2 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:15:23.868217+0000 3 0 2024-09-01T03:05:46.521 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:05:27.563211+0000 34'50 55:151 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:54:33.182204+0000 1 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:38.133241+0000 42'51 53:143 [1,2,0] 1 [1,2,0] 1 42'51 2024-09-01T03:05:38.133145+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:15:36.564830+0000 3 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:22.429246+0000 42'51 53:147 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:25:29.405959+0000 1 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:05:26.070166+0000 36'50 54:143 [2,0,1] 2 [2,0,1] 2 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:20:25.284539+0000 3 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:05:32.162779+0000 36'51 53:137 [1,0,2] 1 [1,0,2] 1 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T13:21:48.156756+0000 4 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 483 0 0 13 36 13 active+clean 2024-09-01T03:05:28.317547+0000 32'49 53:141 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:30:01.784616+0000 1 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:05:33.761026+0000 42'51 53:153 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:17:56.537731+0000 2 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:05:37.176613+0000 32'49 53:117 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:05:37.176567+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:19:33.944518+0000 2 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:05:40.110559+0000 41'51 53:131 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:40.110498+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:26:42.237730+0000 4 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 6 0 0 0 0 2086 0 0 17 36 17 active+clean 2024-09-01T03:05:39.354696+0000 49'53 53:151 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:50:15.117270+0000 4 0 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:05:46.522 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:05:46.523 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:05:46.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-01T03:05:46.531 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:05:46.531 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:05:46.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:05:46.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:05:46.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:05:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:05:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-01T03:05:46.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-01T03:05:47.273 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:05:47.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-01T03:05:47.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-01T03:05:47.783 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T03:05:47.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-01T03:05:47.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-01T03:05:48.293 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T03:05:48.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:05:48.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-01T03:05:48.798 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-09-01T03:05:48.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:05:48.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-01T03:05:49.309 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-09-01T03:05:49.320 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-09-01T03:05:49.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-01T03:05:49.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:05:49.688 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:05:49.688 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:25.242967+0000 36'51 53:151 [1,2,0] 1 [1,2,0] 1 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:33:45.606178+0000 3 0 2024-09-01T03:05:49.688 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:30.144701+0000 42'51 53:153 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:48:21.690397+0000 2 0 2024-09-01T03:05:49.688 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 1932 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:23.045178+0000 38'51 54:143 [2,0,1] 2 [2,0,1] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:10:39.716002+0000 4 0 2024-09-01T03:05:49.688 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1449 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:39.421487+0000 41'50 53:148 [1,0,2] 1 [1,0,2] 1 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:27:27.891740+0000 3 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1449 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:39.108382+0000 34'50 53:118 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:39.108302+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:16:07.213169+0000 3 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 2086 0 0 16 36 16 stale+active+clean 2024-09-01T03:05:36.123976+0000 49'52 53:139 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:36.123925+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:56:06.034548+0000 5 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1449 0 0 16 36 16 stale+active+clean 2024-09-01T03:05:33.760957+0000 42'52 53:154 [1,2,0] 1 [1,2,0] 1 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:11:32.436877+0000 3 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 966 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:34.180660+0000 34'50 53:142 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T09:37:32.050402+0000 2 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 4 0 0 0 0 1932 0 0 16 36 16 stale+active+clean 2024-09-01T03:05:35.137001+0000 39'52 53:146 [1,0,2] 1 [1,0,2] 1 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:14:09.607181+0000 4 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 2898 0 0 16 36 16 stale+active+clean 2024-09-01T03:05:28.167697+0000 41'52 54:157 [2,0,1] 2 [2,0,1] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:18:50.253068+0000 6 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 966 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:24.325712+0000 34'50 53:146 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:49:56.676661+0000 2 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 2415 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:24.103891+0000 43'51 53:143 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:05:19.446048+0000 4 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 483 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:28.519212+0000 38'50 55:141 [0,1,2] 0 [0,1,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:43:48.006893+0000 1 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:23.327391+0000 41'51 53:147 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:42:47.300526+0000 2 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 966 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:21.164363+0000 41'51 54:143 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:44:09.307422+0000 0 0 2024-09-01T03:05:49.689 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 966 0 0 13 36 13 stale+active+clean 2024-09-01T03:05:20.369651+0000 32'49 53:145 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:24:09.248391+0000 2 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1449 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:16.302016+0000 42'50 53:146 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:18:25.490803+0000 2 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:22.112543+0000 41'51 54:144 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:23:48.931660+0000 1 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 0 0 0 1449 0 0 17 36 17 stale+active+clean 2024-09-01T03:05:31.138546+0000 49'53 53:148 [1,0,2] 1 [1,0,2] 1 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:56:24.651716+0000 4 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 1932 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:33.203146+0000 36'50 53:138 [1,2,0] 1 [1,2,0] 1 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:42:34.647837+0000 4 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:27.345972+0000 42'51 53:147 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T13:48:26.672497+0000 2 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 2 0 0 0 0 966 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:27.067998+0000 49'51 54:155 [2,0,1] 2 [2,0,1] 2 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T07:15:23.868217+0000 3 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 483 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:27.563211+0000 34'50 55:151 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:54:33.182204+0000 1 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:38.133241+0000 42'51 53:143 [1,2,0] 1 [1,2,0] 1 42'51 2024-09-01T03:05:38.133145+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:15:36.564830+0000 3 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:22.429246+0000 42'51 53:147 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:25:29.405959+0000 1 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1449 0 0 14 36 14 stale+active+clean 2024-09-01T03:05:26.070166+0000 36'50 54:143 [2,0,1] 2 [2,0,1] 2 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:20:25.284539+0000 3 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 1932 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:32.162779+0000 36'51 53:137 [1,0,2] 1 [1,0,2] 1 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T13:21:48.156756+0000 4 0 2024-09-01T03:05:49.690 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 483 0 0 13 36 13 stale+active+clean 2024-09-01T03:05:28.317547+0000 32'49 53:141 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:30:01.784616+0000 1 0 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1449 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:33.761026+0000 42'51 53:153 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:17:56.537731+0000 2 0 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 966 0 0 13 36 13 stale+active+clean 2024-09-01T03:05:37.176613+0000 32'49 53:117 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:05:37.176567+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:19:33.944518+0000 2 0 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 1932 0 0 15 36 15 stale+active+clean 2024-09-01T03:05:40.110559+0000 41'51 53:131 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:40.110498+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:26:42.237730+0000 4 0 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 6 0 0 0 0 2086 0 0 17 36 17 stale+active+clean 2024-09-01T03:05:39.354696+0000 49'53 53:151 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:50:15.117270+0000 4 0 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:05:49.691 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:05:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:05:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-01T03:05:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:05:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:05:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:05:49.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:05:49.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:05:49.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:05:49.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:05:49.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:05:49.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:05:49.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-01T03:05:49.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:05:49.708 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:05:49.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:05:49.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-01T03:05:49.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:05:49.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:05:49.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:05:49.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:05:49.736 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:49.734+0000 7f5d9d9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:05:49.738 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:49.736+0000 7f5d9d9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:05:49.739 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:49.737+0000 7f5d9d9dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:05:50.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:05:50.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:05:50.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:05:50.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:05:50.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:05:50.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:50.137 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:05:50.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:05:50.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:50.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:05:50.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:05:50.833 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:50.832+0000 7f5d9d9dd640 -1 Falling back to public interface 2024-09-01T03:05:51.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:05:51.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:51.729 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:05:51.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:05:51.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:51.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:05:52.162 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:05:53.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:05:53.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:53.164 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:05:53.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:05:53.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:53.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:05:53.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:05:54.116 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:54.113+0000 7f5d9d9dd640 -1 osd.0 49 log_to_monitors true 2024-09-01T03:05:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:05:54.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:54.561 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:05:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:05:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:54.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:05:55.000 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 68 up_thru 37 down_at 60 last_clean_interval [5,49) [v2:127.0.0.1:6802/2193437468,v1:127.0.0.1:6803/2193437468] [v2:127.0.0.1:6804/2193437468,v1:127.0.0.1:6805/2193437468] exists,up 0bb18a82-93ad-4ae9-8f56-88086e781f9b 2024-09-01T03:05:55.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:05:55.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:05:55.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:05:55.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:05:55.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-01T03:05:55.007 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:05:55.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:05:55.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:05:55.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-01T03:05:55.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:05:55.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:05:55.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:05:55.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:05:55.038 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:55.036+0000 7f33bac13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:05:55.040 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:55.039+0000 7f33bac13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:05:55.041 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:55.040+0000 7f33bac13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:05:55.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:05:55.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:05:55.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:05:55.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:05:55.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:05:55.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:55.453 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:05:55.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:05:55.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:55.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:05:55.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:05:56.399 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:05:56.397+0000 7f33bac13640 -1 Falling back to public interface 2024-09-01T03:05:56.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:05:56.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:56.887 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:05:56.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:05:56.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:56.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:05:57.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:05:58.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:05:58.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:58.332 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:05:58.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:05:58.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:58.333 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:05:58.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:05:59.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:05:59.733 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:05:59.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:05:59.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:05:59.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:05:59.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:06:00.016 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:06:00.014+0000 7f33bac13640 -1 osd.2 49 log_to_monitors true 2024-09-01T03:06:00.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:06:01.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:06:01.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:06:01.128 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:06:01.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:06:01.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:06:01.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:06:01.573 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 73 up_thru 55 down_at 62 last_clean_interval [15,49) [v2:127.0.0.1:6810/1171244852,v1:127.0.0.1:6811/1171244852] [v2:127.0.0.1:6812/1171244852,v1:127.0.0.1:6813/1171244852] exists,up 4fb7a0c5-453b-4a68-91b5-29d07b3434d4 2024-09-01T03:06:01.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:06:01.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:06:01.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:06:01.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:06:01.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-01T03:06:02.243 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-09-01T03:06:02.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:06:02.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-01T03:06:02.988 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:06:03.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:06:03.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:06:03.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:06:03.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:06:03.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:06:03.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:06:03.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:06:03.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:06:03.528 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:06:03.528 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:06:03.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:06:03.528 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:06:03.528 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:06:03.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776132 2024-09-01T03:06:03.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776132 2024-09-01T03:06:03.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776132' 2024-09-01T03:06:03.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:06:03.780 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:06:03.869 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T03:06:03.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T03:06:03.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T03:06:03.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T03:06:03.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:06:03.871 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:06:04.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612611 2024-09-01T03:06:04.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612611 2024-09-01T03:06:04.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776132 2-313532612611' 2024-09-01T03:06:04.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:06:04.085 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-292057776132 2024-09-01T03:06:04.085 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:06:04.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:06:04.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-292057776132 2024-09-01T03:06:04.086 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:06:04.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776132 2024-09-01T03:06:04.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 292057776132' 2024-09-01T03:06:04.088 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 292057776132 2024-09-01T03:06:04.088 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:06:04.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776130 -lt 292057776132 2024-09-01T03:06:04.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:06:05.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:06:05.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:06:05.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776133 -lt 292057776132 2024-09-01T03:06:05.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:06:05.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612611 2024-09-01T03:06:05.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:06:05.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:06:05.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612611 2024-09-01T03:06:05.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:06:05.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612611 2024-09-01T03:06:05.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-09-01T03:06:05.998 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 313532612611 2024-09-01T03:06:05.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:06:06.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-01T03:06:06.471 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:06:06.472 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:06.472 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:07.029 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:07.416 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:07.416 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:07.417 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:07.417 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:07.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:07.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T03:06:07.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T03:06:07.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T03:06:07.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:07.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T03:06:08.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:08.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:08.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:08.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:08.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:08.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:08.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:08.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:08.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:08.453 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:08.453 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:08.453 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:09.011 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:09.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:09.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:09.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:09.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:09.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:09.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:09.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:09.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T03:06:09.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:09.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T03:06:09.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:09.743 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:10.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:10.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:10.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:10.124 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:10.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:10.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:10.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:10.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:10.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:10.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:10.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:10.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:10.648 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:11.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:11.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:11.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T03:06:11.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:11.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T03:06:11.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:11.594 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:11.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:11.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:11.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:11.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:12.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:12.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:12.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:12.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:12.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:12.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:12.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:12.512 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:12.512 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T03:06:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T03:06:13.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:13.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:13.871 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:13.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:13.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:13.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:13.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:13.872 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:14.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:14.265 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:14.265 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:14.265 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:14.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:14.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:14.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:14.804 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:14.804 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:15.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:15.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:15.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T03:06:15.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:15.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T03:06:16.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:16.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:16.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:16.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:16.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:16.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:16.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:16.930 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:17.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:17.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:17.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:17.326 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:17.849 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T03:06:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:18.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T03:06:21.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:21.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:21.619 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:21.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:21.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:21.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:21.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:21.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:22.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:22.012 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:22.013 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:22.013 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:22.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:22.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:22.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:23.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:23.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:23.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T03:06:23.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:23.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T03:06:29.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:29.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:29.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:29.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:29.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:29.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:29.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:29.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:29.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:29.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:29.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:29.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:30.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:30.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:30.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:31.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:31.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:31.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T03:06:31.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:31.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:38.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:38.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:38.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:38.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:38.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:39.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:39.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:39.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:39.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:39.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:39.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:39.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:39.233 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:39.233 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:39.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:39.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:39.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T03:06:39.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "09c63a9e-4a53-45ab-9d1d-340c109734e1", 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T03:06:40.303114+0000", 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T03:06:40.305 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:06:40.306 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:06:40.307 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 32 pgs degraded", 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 32 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.308 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.309 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.310 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.311 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.312 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.312 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.312 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.312 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.312 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.313 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.314 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.315 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:06:40.316 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.317 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:03:07.028296Z", 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:03:07.028296Z", 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T03:06:40.318 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T03:06:40.319 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T03:06:40.320 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.321 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.322 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 80, 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:03:07.871443+0000", 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:06:03.515281+0000", 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T03:06:00.316016+0000", 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T03:06:02.651015+0000", 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T03:06:40.323 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T03:06:40.324 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T03:04:54.816253+0000", 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T03:06:40.325 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "on", 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 32, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 24, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 32, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 32, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 32, 2024-09-01T03:06:40.326 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "78", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "33", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-09-01T03:06:40.327 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T03:06:40.328 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T03:06:40.329 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 1.5800000429153442, 2024-09-01T03:06:40.330 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 1.5800000429153442, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 1.059999942779541, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 1.059999942779541, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:06:40.331 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "0bb18a82-93ad-4ae9-8f56-88086e781f9b", 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 49, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 68, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 78, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 60, 2024-09-01T03:06:40.332 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.333 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.334 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.335 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.336 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/2193437468", 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/2193437468", 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2193437468", 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2193437468", 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "427ee48c-66f3-493b-b6c3-1c333687da39", 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T03:06:40.337 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 57, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 61, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.338 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.339 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.340 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.341 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1570722299", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1570722299", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1570722299", 2024-09-01T03:06:40.342 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1570722299", 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "4fb7a0c5-453b-4a68-91b5-29d07b3434d4", 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-09-01T03:06:40.343 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 49, 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 73, 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 79, 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 62, 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.344 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.345 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.346 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.347 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1171244852", 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1171244852", 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1171244852", 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1171244852", 2024-09-01T03:06:40.348 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:05:47.036520+0000", 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:06:40.349 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:03:41.677733+0000", 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:05:47.612081+0000", 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:04:16.349879+0000", 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.350 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:05:48.123028+0000", 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:04:51.831095+0000", 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T03:06:40.351 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T03:06:40.352 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T03:06:40.353 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/2193437468,v1:127.0.0.1:6805/2193437468]", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:06:40.354 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:06:40.355 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:03:13.911371Z", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/2193437468,v1:127.0.0.1:6803/2193437468]", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2193437468,v1:127.0.0.1:6809/2193437468]", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2193437468,v1:127.0.0.1:6807/2193437468]", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:06:40.356 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.357 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1570722299,v1:127.0.0.1:6813/1570722299]", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:06:40.358 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:03:48.125402Z", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:06:40.359 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1570722299,v1:127.0.0.1:6811/1570722299]", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1570722299,v1:127.0.0.1:6817/1570722299]", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1570722299,v1:127.0.0.1:6815/1570722299]", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:06:40.360 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1171244852,v1:127.0.0.1:6813/1171244852]", 2024-09-01T03:06:40.361 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:06:40.362 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:04:23.468184Z", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:06:40.363 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1171244852,v1:127.0.0.1:6811/1171244852]", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1171244852,v1:127.0.0.1:6817/1171244852]", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1171244852,v1:127.0.0.1:6815/1171244852]", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:06:40.364 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [ 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:06:40.365 INFO:tasks.workunit.client.0.smithi079.stdout: "floor": 36 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 69 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 73 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.366 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 80, 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:06:40.367 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:06:40.368 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.369 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T03:06:40.370 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.371 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.371 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T03:06:40.371 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T03:06:40.371 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.371 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.371 INFO:tasks.workunit.client.0.smithi079.stderr:report 3097571970 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T03:06:40.372 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.373 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.374 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:06:40.375 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.376 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:06:40.377 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.378 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:06:40.379 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T03:06:40.380 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T03:06:40.381 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T03:03:07:846833+0000", 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T03:06:40.382 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T03:06:40.383 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-09-01T03:06:40.384 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 32, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48512, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:06:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 57, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 57, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:06:40.386 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:06:40.387 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:06:40.388 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 467, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 467, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:06:40.389 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 84408, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2920, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81397, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488392, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036113408, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2990080, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 723984, 2024-09-01T03:06:40.390 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83351396 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.391 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 116, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 116, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 116000000, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 116000000 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:06:40.392 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 84408, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2920, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81397, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488392, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036113408, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:06:40.393 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2990080, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 723984, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83351396 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:06:40.394 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 116, 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 116, 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 116000000, 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 116000000 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T03:06:40.395 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48512, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:06:40.396 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 57, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 57, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:06:40.397 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:06:40.398 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 143276, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:06:40.399 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 467, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 467, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 292057776139 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.400 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:06:40.401 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672986 2024-09-01T03:06:40.401 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.401 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.404 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:06:40.404 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 313532612618 2024-09-01T03:06:40.404 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.404 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 32 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 17, 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.405 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 15, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.406 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:05:52.591940+0000", 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 95, 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T03:06:40.407 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 442, 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 600, 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:06:40.408 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:06:40.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T03:06:40.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-01T03:06:40.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-01T03:06:40.409 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-09-01T03:06:40.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-01T03:06:40.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:06:40.710 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 80:114 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 80:128 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 80:112 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 80:116 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 80:97 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 80:135 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 80:120 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 80:114 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 80:127 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 80:134 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 80:102 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 80:126 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 80:120 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 80:129 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 80:118 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 80:89 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 80:131 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 80:119 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 41'51 80:113 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 80:102 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 80:132 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 80:101 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.712 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 80:130 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 80:120 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 80:127 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 80:100 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 80:139 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 80:89 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 80:127 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 80:133 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 80:116 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 80:128 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:06:40.713 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:06:40.714 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:06:40.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:06:40.862 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:06:40.863 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:06:40.863 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:06:40.863 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:06:40.863 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:06:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:06:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:06:40.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:06:40.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:06:40.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:06:40.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:06:41.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:06:41.340 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:06:41.340 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:06:41.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:06:41.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:06:41.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:06:41.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776142 2024-09-01T03:06:41.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776142 2024-09-01T03:06:41.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776142' 2024-09-01T03:06:41.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:06:41.461 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:06:41.548 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T03:06:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T03:06:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T03:06:41.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T03:06:41.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:06:41.551 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:06:41.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612621 2024-09-01T03:06:41.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612621 2024-09-01T03:06:41.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776142 2-313532612621' 2024-09-01T03:06:41.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:06:41.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-292057776142 2024-09-01T03:06:41.672 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:06:41.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:06:41.673 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-292057776142 2024-09-01T03:06:41.673 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:06:41.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776142 2024-09-01T03:06:41.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 292057776142' 2024-09-01T03:06:41.674 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 292057776142 2024-09-01T03:06:41.675 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:06:42.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776140 -lt 292057776142 2024-09-01T03:06:42.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:06:43.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:06:43.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:06:43.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776142 -lt 292057776142 2024-09-01T03:06:43.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:06:43.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612621 2024-09-01T03:06:43.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:06:43.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:06:43.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612621 2024-09-01T03:06:43.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:06:43.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612621 2024-09-01T03:06:43.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612621' 2024-09-01T03:06:43.555 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 313532612621 2024-09-01T03:06:43.555 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:06:43.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612621 -lt 313532612621 2024-09-01T03:06:43.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:06:43.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:43.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:44.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-01T03:06:44.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:44.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:44.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:44.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:44.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:44.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:44.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:44.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:44.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:44.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:44.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:45.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:45.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T03:06:45.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T03:06:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T03:06:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:45.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T03:06:45.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:45.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:45.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:45.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:45.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:45.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:45.538 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:45.538 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:45.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:45.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:45.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:45.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:46.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:46.445 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:46.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:46.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:46.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T03:06:46.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:46.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:47.168 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:47.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:47.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:47.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:47.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:48.075 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:48.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:48.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:48.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T03:06:48.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:48.607 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T03:06:49.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:49.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:49.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:49.391 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:49.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:49.392 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:49.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:49.922 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:49.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:50.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:50.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:50.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T03:06:50.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:50.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T03:06:51.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:51.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:51.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:51.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:51.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:51.638 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:52.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:52.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:52.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:52.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:52.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:52.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:52.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:52.203 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:52.203 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:52.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:52.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:52.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T03:06:52.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:52.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T03:06:54.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:54.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:54.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:54.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:54.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:54.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:54.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:54.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:54.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:54.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:54.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:54.738 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:06:55.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:06:55.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:06:55.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:06:55.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:06:55.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:06:55.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:06:55.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:06:55.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:06:55.302 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:06:55.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:06:55.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:06:55.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T03:06:55.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:06:55.839 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T03:06:59.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:06:59.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:06:59.041 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:06:59.041 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:06:59.041 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:06:59.041 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:06:59.042 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:06:59.042 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:06:59.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:06:59.432 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:06:59.432 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:06:59.432 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:07:00.018 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:07:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:07:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:07:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T03:07:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:07:00.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T03:07:06.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:07:06.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:07:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:07:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:07:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:07:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:07:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:07:06.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:07:07.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:07:07.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:07:07.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:07:07.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:07:07.867 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:07:07.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:07:07.868 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:07:08.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:07:08.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:07:08.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T03:07:08.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:07:08.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T03:07:15.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:07:15.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:07:15.728 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:07:15.728 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:07:15.728 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:07:15.728 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:07:15.728 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:07:15.728 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:07:16.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:07:16.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:07:16.127 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:07:16.127 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:07:16.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-01T03:07:16.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:07:16.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:07:16.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:07:16.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:07:16.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:07:16.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:07:16.659 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:07:16.659 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:07:17.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:07:17.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:07:17.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T03:07:17.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T03:07:17.701 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "09c63a9e-4a53-45ab-9d1d-340c109734e1", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T03:07:17.699725+0000", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T03:07:17.702 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.703 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 64 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.704 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.3 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.705 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.4 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.5 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.6 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.7 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.706 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.8 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.9 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.707 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.e is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.708 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.11 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.12 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.13 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.709 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.14 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.15 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.16 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.17 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.710 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.18 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.19 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.711 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.712 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-01T03:07:17.712 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.712 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.712 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.712 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.712 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1e is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.1f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:07:17.713 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:07:17.714 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:03:07.028296Z", 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:03:07.028296Z", 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T03:07:17.715 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T03:07:17.716 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.717 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.718 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T03:07:17.719 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 80, 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:03:07.871443+0000", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:06:03.515281+0000", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T03:06:00.316016+0000", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T03:06:02.651015+0000", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T03:07:17.720 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 9, 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T03:07:17.721 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T03:04:54.816253+0000", 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 1, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool", 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 0, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T03:07:17.722 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "on", 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 32, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 24, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 32, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 32, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 32, 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T03:07:17.723 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "78", 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "33", 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "selfmanaged", 2024-09-01T03:07:17.724 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 0, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 0, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [], 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T03:07:17.725 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "", 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T03:07:17.726 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T03:07:17.727 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {}, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "read_balance": { 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "score_type": "Fair distribution", 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "score_acting": 1.5800000429153442, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "score_stable": 1.5800000429153442, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_score": 0.67000001668930054, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_acting": 1.059999942779541, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "raw_score_stable": 1.059999942779541, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity_weighted": 1, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity": 1, 2024-09-01T03:07:17.728 INFO:tasks.workunit.client.0.smithi079.stdout: "average_primary_affinity_weighted": 1 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "0bb18a82-93ad-4ae9-8f56-88086e781f9b", 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-09-01T03:07:17.729 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 49, 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 68, 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 78, 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 60, 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.730 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.731 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.732 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.733 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 2193437468 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/2193437468", 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/2193437468", 2024-09-01T03:07:17.734 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2193437468", 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2193437468", 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "427ee48c-66f3-493b-b6c3-1c333687da39", 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:07:17.735 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 57, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 61, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.736 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.737 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:07:17.738 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.739 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1570722299 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1570722299", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1570722299", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1570722299", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1570722299", 2024-09-01T03:07:17.740 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "4fb7a0c5-453b-4a68-91b5-29d07b3434d4", 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 49, 2024-09-01T03:07:17.741 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 73, 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 79, 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 62, 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.742 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.743 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.744 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:07:17.745 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1171244852 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/1171244852", 2024-09-01T03:07:17.746 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/1171244852", 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1171244852", 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1171244852", 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:05:47.036520+0000", 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:07:17.747 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:03:41.677733+0000", 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:05:47.612081+0000", 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:07:17.748 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:04:16.349879+0000", 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:05:48.123028+0000", 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:04:51.831095+0000", 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.749 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:07:17.750 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T03:07:17.751 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/2193437468,v1:127.0.0.1:6805/2193437468]", 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:07:17.752 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:07:17.753 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:03:13.911371Z", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:07:17.754 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/2193437468,v1:127.0.0.1:6803/2193437468]", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2193437468,v1:127.0.0.1:6809/2193437468]", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2193437468,v1:127.0.0.1:6807/2193437468]", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:07:17.755 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1570722299,v1:127.0.0.1:6813/1570722299]", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:07:17.756 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:07:17.757 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:03:48.125402Z", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:07:17.758 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1570722299,v1:127.0.0.1:6811/1570722299]", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1570722299,v1:127.0.0.1:6817/1570722299]", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1570722299,v1:127.0.0.1:6815/1570722299]", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T03:07:17.759 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/1171244852,v1:127.0.0.1:6813/1171244852]", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:07:17.760 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:07:17.761 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:04:23.468184Z", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:07:17.762 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/1171244852,v1:127.0.0.1:6811/1171244852]", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1171244852,v1:127.0.0.1:6817/1171244852]", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1171244852,v1:127.0.0.1:6815/1171244852]", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:07:17.763 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [ 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:07:17.764 INFO:tasks.workunit.client.0.smithi079.stdout: "floor": 36 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 69 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:07:17.765 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 73 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stderr:report 4060871949 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 80, 2024-09-01T03:07:17.766 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.767 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T03:07:17.768 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T03:07:17.769 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T03:07:17.770 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.771 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:07:17.772 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.773 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:07:17.774 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T03:07:17.775 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.776 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.777 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:07:17.778 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:07:17.779 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:07:17.780 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.781 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T03:07:17.782 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 0, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T03:07:17.783 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T03:03:07:846833+0000", 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T03:07:17.784 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.785 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.786 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 32, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48183, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:07:17.787 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 58, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 58, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:07:17.788 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:07:17.789 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:07:17.790 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 468, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 468, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-09-01T03:07:17.791 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:07:17.792 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 84408, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2920, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81397, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488392, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036113408, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2990080, 2024-09-01T03:07:17.793 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 723326, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83351396 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:07:17.794 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:07:17.795 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:07:17.795 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.795 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:07:17.795 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 116, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 116, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 116000000, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 116000000 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 96, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:07:17.796 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 84408, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2920, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81397, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314488392, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036113408, 2024-09-01T03:07:17.797 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2990080, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 723326, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10396, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83351396 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:07:17.798 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 116, 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 116, 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 116000000, 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 116000000 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.799 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:07:17.800 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:07:17.800 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.800 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.800 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T03:07:17.800 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.800 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 32, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48183, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 0, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:07:17.801 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:07:17.802 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:07:17.802 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:07:17.802 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 58, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 58, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:07:17.803 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:07:17.804 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:07:17.805 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1220608, 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 142618, 2024-09-01T03:07:17.806 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 468, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 468, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 64, 2024-09-01T03:07:17.807 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 64, 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 292057776149 2024-09-01T03:07:17.808 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672986 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 313532612628 2024-09-01T03:07:17.809 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 32 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T03:07:17.810 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 17, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 15, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 32, 2024-09-01T03:07:17.811 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T03:07:17.812 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 7, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:07:00.607667+0000", 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 114, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 510, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 687, 2024-09-01T03:07:17.813 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T03:07:17.814 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T03:07:17.814 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:07:17.814 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:07:17.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T03:07:17.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-01T03:07:17.815 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-09-01T03:07:17.815 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:18.103 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:18.103 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 80:114 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.103 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 80:128 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:07:18.103 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 80:112 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 80:116 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 80:97 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 80:135 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 80:120 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 80:114 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 80:127 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 80:134 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 80:102 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 80:126 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 80:120 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 80:129 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 80:118 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:07:18.104 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 80:89 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 80:131 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 80:119 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1603 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 80'52 80:114 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 80:102 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 80:132 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 80:101 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 80:130 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 80:120 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 80:127 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 80:100 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 80:139 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 80:89 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.105 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 80:127 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:07:18.106 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 80:133 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:07:18.106 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 80:116 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.106 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 80:128 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:07:18.106 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:18.106 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:18.106 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:18.115 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:07:18.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-01T03:07:18.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok set_recovery_delay 100000 2024-09-01T03:07:18.196 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:07:18.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:07:18.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T03:07:18.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-01T03:07:18.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok set_recovery_delay 100000 2024-09-01T03:07:18.285 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:07:18.292 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-09-01T03:07:18.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-01T03:07:18.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 80:114 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 80:128 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 80:112 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 80:116 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 80:97 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 80:135 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 80:120 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 80:114 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.662 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 80:127 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 80:134 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 80:102 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 80:126 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 80:120 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 80:129 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 80:118 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 80:89 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 80:131 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 80:119 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1603 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 80'52 80:114 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 80:102 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 80:132 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:07:18.663 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 80:101 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 80:130 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 80:120 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 80:127 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 80:100 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 80:139 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 80:89 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 80:127 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 80:133 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 80:116 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 80:128 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:18.664 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:18.665 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:18.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-01T03:07:19.343 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-09-01T03:07:19.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-01T03:07:19.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:07:19.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:07:19.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:07:19.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:07:19.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:07:19.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:07:19.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:07:19.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:07:19.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:07:19.360 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:07:19.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:07:19.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:07:19.361 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T03:07:19.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:07:19.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:07:19.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:07:19.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:07:19.392 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:19.391+0000 7f41ece5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:07:19.402 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:19.400+0000 7f41ece5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:07:19.403 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:19.402+0000 7f41ece5e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:07:19.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-01T03:07:19.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-01T03:07:20.740 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:20.739+0000 7f41ece5e640 -1 Falling back to public interface 2024-09-01T03:07:23.704 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:23.702+0000 7f41ece5e640 -1 osd.1 58 log_to_monitors true 2024-09-01T03:07:24.828 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-09-01T03:07:24.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-01T03:07:24.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:25.207 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 82:117 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:19.717600+0000 0 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 82:131 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 82:115 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 83:122 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.191451+0000 0 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 82:100 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:23.629325+0000 0 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 82:138 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 82:123 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:20.708186+0000 0 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 82:117 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 83:133 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:29.186885+0000 0 0 2024-09-01T03:07:25.208 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 82:137 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 83:108 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 82:129 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 83:126 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 83:135 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 82:121 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 83:137 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 82:122 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1603 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 80'52 82:117 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 82:105 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:22.671122+0000 0 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 83:138 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:07:25.209 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 83:107 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 83:136 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 83:126 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.220322+0000 0 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 82:130 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 83:106 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.221518+0000 0 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 83:145 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.212029+0000 0 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 82:130 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 82:136 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 82:119 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:21.658879+0000 0 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 83:134 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:25.210 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:25.211 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-01T03:07:25.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok set_recovery_delay 100000 2024-09-01T03:07:25.301 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:07:25.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-01T03:07:26.252 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-09-01T03:07:26.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-01T03:07:26.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:07:26.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:07:26.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:07:26.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:07:26.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:07:26.265 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:07:26.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:07:26.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:07:26.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:07:26.956 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 68 up_thru 85 down_at 60 last_clean_interval [5,49) [v2:127.0.0.1:6802/2193437468,v1:127.0.0.1:6803/2193437468] [v2:127.0.0.1:6804/2193437468,v1:127.0.0.1:6805/2193437468] exists,up 0bb18a82-93ad-4ae9-8f56-88086e781f9b 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:07:26.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:07:26.959 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:07:26.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:07:26.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:07:27.428 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 85 up_thru 85 down_at 61 last_clean_interval [10,58) [v2:127.0.0.1:6818/1685115119,v1:127.0.0.1:6819/1685115119] [v2:127.0.0.1:6820/1685115119,v1:127.0.0.1:6821/1685115119] exists,up 427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:07:27.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:07:27.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:07:27.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:07:27.430 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:07:27.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:07:27.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:07:27.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:07:27.853 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 73 up_thru 85 down_at 62 last_clean_interval [15,49) [v2:127.0.0.1:6810/1171244852,v1:127.0.0.1:6811/1171244852] [v2:127.0.0.1:6812/1171244852,v1:127.0.0.1:6813/1171244852] exists,up 4fb7a0c5-453b-4a68-91b5-29d07b3434d4 2024-09-01T03:07:27.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:07:27.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:07:27.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:07:27.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 83:119 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.620812+0000 0 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 83:133 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 83:117 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 83:122 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.191451+0000 0 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 83:102 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 83:140 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 83:125 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.644952+0000 0 0 2024-09-01T03:07:28.227 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 83:119 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:24.656807+0000 0 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 83:133 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:29.186885+0000 0 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 83:139 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 83:108 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 83:131 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 83:126 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 83:135 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 83:123 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 83:137 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 83:124 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1603 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 80'52 83:119 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:07:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 83:107 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.617896+0000 0 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 83:138 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 83:107 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 83:136 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 83:126 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.220322+0000 0 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 83:132 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 83:106 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.221518+0000 0 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 83:145 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.212029+0000 0 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 83:132 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:07:28.229 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 83:138 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:07:28.230 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 83:121 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.624063+0000 0 0 2024-09-01T03:07:28.230 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 83:134 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:07:28.230 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:28.230 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:28.230 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:28.238 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-09-01T03:07:28.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-01T03:07:28.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:28.605 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 83:119 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.620812+0000 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 83:133 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 83:117 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 83:122 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.191451+0000 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 83:102 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 83:140 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 83:125 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.644952+0000 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 83:119 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:24.656807+0000 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 83:133 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:29.186885+0000 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 83:139 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 83:108 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.606 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 83:131 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 83:126 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 83:135 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 83:123 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 83:137 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 83:124 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1603 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 80'52 83:119 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 83:107 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.617896+0000 0 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 83:138 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 83:107 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 83:136 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 83:126 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.220322+0000 0 0 2024-09-01T03:07:28.607 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 83:132 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 83:106 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.221518+0000 0 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 83:145 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.212029+0000 0 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 83:132 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 83:138 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 83:121 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.624063+0000 0 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 83:134 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:28.608 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:28.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-01T03:07:28.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-01T03:07:28.800 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-09-01T03:07:28.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111112+0000 36'51 83:119 [2,0] 2 [2,0] 2 36'51 2024-09-01T03:05:25.242900+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.620812+0000 0 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111834+0000 42'51 83:133 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:51.720094+0000 2 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112046+0000 38'51 83:117 [2,0] 2 [2,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:19:15.746176+0000 4 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600303+0000 41'50 83:122 [0,2] 0 [0,2] 0 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.191451+0000 0 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.112570+0000 34'50 83:102 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:24.605226+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112786+0000 41'51 83:140 [2,0] 2 [2,0] 2 40'50 2024-09-01T03:05:25.056445+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:13:23.513000+0000 4 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111787+0000 42'52 83:125 [2,0] 2 [2,0] 2 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.644952+0000 0 0 2024-09-01T03:07:29.170 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.018812+0000 34'50 83:119 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:34.180609+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:24.656807+0000 0 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 5 0 0 2189 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:03.599135+0000 39'52 83:133 [0,2] 0 [0,2] 0 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:29.186885+0000 0 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 6 0 0 2898 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.112214+0000 41'52 83:139 [2,0] 2 [2,0] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:08:36.022233+0000 6 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 34'50 83:108 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111035+0000 43'51 83:131 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:16:40.683647+0000 4 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599127+0000 38'50 83:126 [0,2] 0 [0,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:22:31.874241+0000 1 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600819+0000 41'51 83:135 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:01:26.327694+0000 2 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 2 0 0 966 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111765+0000 41'51 83:123 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:40.456299+0000 0 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599825+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 3 0 0 1449 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.599137+0000 42'50 83:137 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:30:15.031403+0000 2 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111385+0000 41'51 83:124 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:32:33.810775+0000 1 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 0 4 0 0 1603 0 0 16 36 16 active+undersized+degraded 2024-09-01T03:06:04.111680+0000 80'52 83:119 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:17:46.076294+0000 2 0 2024-09-01T03:07:29.171 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:04.111422+0000 36'50 83:107 [2,0] 2 [2,0] 2 36'50 2024-09-01T03:05:33.203095+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:28.617896+0000 0 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600012+0000 42'51 83:138 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:55:09.465527+0000 2 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 3 0 0 1223 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600053+0000 34'50 83:107 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 1 0 0 483 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.600671+0000 34'50 83:136 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:48:39.208065+0000 1 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 3 0 0 1706 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600058+0000 42'51 83:126 [0,2] 0 [0,2] 0 0'0 2024-09-01T03:04:54.895494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:25.220322+0000 0 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.111068+0000 42'51 83:132 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:52:27.435261+0000 1 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 2 0 0 1222 0 0 14 36 14 active+undersized+degraded 2024-09-01T03:06:03.568159+0000 36'50 83:106 [0,2] 0 [0,2] 0 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:26.221518+0000 0 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 4 0 0 1932 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.568179+0000 36'51 83:145 [0,2] 0 [0,2] 0 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:02:21.370304+0000 4 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 1 0 0 739 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:03.599199+0000 32'49 83:95 [0,2] 0 [0,2] 0 32'49 2024-09-01T03:05:28.317494+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.212029+0000 0 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 3 0 0 1449 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112426+0000 42'51 83:132 [2,0] 2 [2,0] 2 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T06:40:57.752499+0000 2 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 2 0 0 966 0 0 13 36 13 active+undersized+degraded 2024-09-01T03:06:04.112934+0000 32'49 83:138 [2,0] 2 [2,0] 2 32'49 2024-09-01T03:05:24.104787+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:29:37.936359+0000 2 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 3 0 3 0 0 1705 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:04.112171+0000 41'51 83:121 [2,0] 2 [2,0] 2 41'51 2024-09-01T03:05:25.526457+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 user requested scrub scheduled @ 2024-09-01T03:07:27.624063+0000 0 0 2024-09-01T03:07:29.172 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 0 5 0 0 2415 0 0 15 36 15 active+undersized+degraded 2024-09-01T03:06:03.600509+0000 42'51 83:134 [0,2] 0 [0,2] 0 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:07:56.306526+0000 4 0 2024-09-01T03:07:29.173 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:29.173 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:29.173 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:29.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-01T03:07:29.181 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:07:29.181 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:07:29.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:07:29.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:07:29.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:07:29.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:07:29.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.22149.out 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.22149.out 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-01T03:07:29.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.22149.out 2024-09-01T03:07:30.596 INFO:tasks.workunit.client.0.smithi079.stderr:Exporting 1.1b info 1.1b( v 42'51 lc 0'0 (32'36,42'51] local-lis/les=85/87 n=5 ec=33/23 lis/c=85/52 les/c/f=87/53/0 sis=85) 2024-09-01T03:07:30.596 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-01T03:07:30.602 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-01T03:07:30.603 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-01T03:07:30.609 INFO:tasks.workunit.client.0.smithi079.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-01T03:07:30.610 INFO:tasks.workunit.client.0.smithi079.stderr:Export successful 2024-09-01T03:07:30.610 INFO:tasks.workunit.client.0.smithi079.stdout: marking collection for removal 2024-09-01T03:07:30.610 INFO:tasks.workunit.client.0.smithi079.stdout:setting '_remove' omap key 2024-09-01T03:07:30.610 INFO:tasks.workunit.client.0.smithi079.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-01T03:07:30.796 INFO:tasks.workunit.client.0.smithi079.stderr:Remove successful 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.22149.out 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-01T03:07:31.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.22149.out 2024-09-01T03:07:32.243 INFO:tasks.workunit.client.0.smithi079.stdout:get_pg_num_history pg_num_history pg_num_history(e87 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-09-01T03:07:32.243 INFO:tasks.workunit.client.0.smithi079.stdout:Importing pgid 1.1b 2024-09-01T03:07:32.243 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-01T03:07:32.243 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:07:32.308 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-01T03:07:32.308 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:07:32.342 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-01T03:07:32.342 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:07:32.375 INFO:tasks.workunit.client.0.smithi079.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-01T03:07:32.375 INFO:tasks.workunit.client.0.smithi079.stdout:snapset 0=[]:{} 2024-09-01T03:07:32.408 INFO:tasks.workunit.client.0.smithi079.stdout:write_pg epoch 87 info 1.1b( v 42'51 lc 0'0 (32'36,42'51] local-lis/les=85/87 n=5 ec=33/23 lis/c=85/52 les/c/f=87/53/0 sis=85) 2024-09-01T03:07:32.442 INFO:tasks.workunit.client.0.smithi079.stdout:Import successful 2024-09-01T03:07:32.958 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-09-01T03:07:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-01T03:07:32.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:28.623160+0000 36'51 87:140 [2,0,1] 2 [2,0,1] 2 36'51 2024-09-01T03:07:28.623114+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:28:57.283799+0000 3 0 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:26.949619+0000 42'51 87:186 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:46:56.628868+0000 2 0 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:26.940953+0000 38'51 87:130 [2,1,0] 2 [2,1,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:43:33.368759+0000 4 0 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1223 0 0 14 36 14 active+clean 2024-09-01T03:07:26.945951+0000 41'50 87:163 [1,0,2] 1 [1,0,2] 1 41'50 2024-09-01T03:05:29.156386+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:26.948681+0000 34'50 87:130 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:39.108302+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:41:32.384433+0000 3 0 2024-09-01T03:07:33.337 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 1 1 0 0 2415 0 0 15 36 15 active+recovery_wait+degraded 2024-09-01T03:07:26.948734+0000 41'51 87:141 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:36.123925+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 4 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 5 0 0 0 0 2189 0 0 16 36 16 active+clean 2024-09-01T03:07:26.948184+0000 42'52 87:154 [1,2,0] 1 [1,2,0] 1 42'52 2024-09-01T03:05:30.091810+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:07:28.705161+0000 34'50 87:149 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:07:28.705116+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T05:47:59.796232+0000 2 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 6 0 0 0 0 2672 0 0 17 36 17 active+clean 2024-09-01T03:07:26.946585+0000 87'53 87:149 [1,0,2] 1 [1,0,2] 1 39'52 2024-09-01T03:05:35.136955+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 2898 0 0 16 36 16 active+clean 2024-09-01T03:07:26.942602+0000 41'52 87:152 [2,0,1] 2 [2,0,1] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:10:31.530722+0000 6 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 3 0 0 0 0 1223 0 0 14 36 14 active+clean 2024-09-01T03:07:26.949532+0000 34'50 87:126 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:24.325655+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:07:26.948904+0000 43'51 87:133 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:02:48.881405+0000 4 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:07:26.941448+0000 38'50 87:140 [0,1,2] 0 [0,1,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T13:23:50.373556+0000 1 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:26.946571+0000 41'51 87:131 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:59:15.050084+0000 2 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 966 0 0 15 36 15 active+clean 2024-09-01T03:07:26.943812+0000 41'51 87:136 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:02:20.397126+0000 0 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 1 0 0 0 0 739 0 0 13 36 13 active+clean 2024-09-01T03:07:26.947644+0000 32'49 87:115 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:20.369595+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:26.950018+0000 42'50 87:133 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:43:34.888018+0000 2 0 2024-09-01T03:07:33.338 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:26.943739+0000 41'51 87:137 [2,1,0] 2 [2,1,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:29:11.985684+0000 1 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 1 1 0 0 1603 0 0 16 36 16 active+recovery_wait+degraded 2024-09-01T03:07:26.950018+0000 80'52 87:178 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 2 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 1932 0 0 14 36 14 active+clean 2024-09-01T03:07:27.669991+0000 36'50 87:128 [2,0,1] 2 [2,0,1] 2 36'50 2024-09-01T03:07:27.669943+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T08:05:37.068010+0000 4 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:26.906581+0000 42'51 87:152 [0,2,1] 0 [0,2,1] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T13:12:21.235459+0000 2 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 1 0 0 1223 0 0 14 36 14 active+recovering+degraded 2024-09-01T03:07:26.949265+0000 34'50 87:120 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 no scrub is scheduled 0 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:07:26.941108+0000 34'50 87:150 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:56:01.282773+0000 1 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1706 0 0 15 36 15 active+clean 2024-09-01T03:07:26.949236+0000 42'51 87:152 [1,0,2] 1 [1,0,2] 1 42'51 2024-09-01T03:05:38.133145+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:26.943233+0000 42'51 87:145 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:31:20.830089+0000 1 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 2 0 0 0 0 1222 0 0 14 36 14 active+clean 2024-09-01T03:07:26.947331+0000 36'50 87:101 [1,0,2] 1 [1,0,2] 1 36'50 2024-09-01T03:05:26.070111+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 queued for scrub 0 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:26.947095+0000 36'51 87:131 [1,0,2] 1 [1,0,2] 1 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:45:48.334448+0000 4 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 483 0 0 13 36 13 active+clean 2024-09-01T03:07:27.181553+0000 32'49 87:117 [0,1,2] 0 [0,1,2] 0 32'49 2024-09-01T03:07:27.181484+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T04:21:52.036267+0000 1 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:26.949825+0000 42'51 87:153 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:12:21.976545+0000 2 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:07:27.735372+0000 32'49 87:134 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:07:27.735324+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:56:05.581390+0000 2 0 2024-09-01T03:07:33.339 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:26.946265+0000 41'51 87:145 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:40.110498+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T05:11:44.440102+0000 4 0 2024-09-01T03:07:33.340 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 1 1 0 0 2415 0 0 15 36 15 active+recovery_wait+degraded 2024-09-01T03:07:26.946849+0000 42'51 87:163 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 4 0 2024-09-01T03:07:33.340 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:33.340 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:33.340 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:07:33.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:07:33.352 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:07:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:07:33.354 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:07:33.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:07:33.388 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:33.384+0000 7f21c761e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:07:33.388 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:33.386+0000 7f21c761e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:07:33.389 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:33.387+0000 7f21c761e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:07:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:07:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:07:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:07:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:07:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:07:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:07:33.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:07:33.746 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:07:33.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:07:33.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:07:34.174 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 85 up_thru 85 down_at 61 last_clean_interval [10,58) [v2:127.0.0.1:6818/1685115119,v1:127.0.0.1:6819/1685115119] [v2:127.0.0.1:6820/1685115119,v1:127.0.0.1:6821/1685115119] exists,up 427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:07:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:07:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:07:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:07:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-01T03:07:34.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:07:34.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:07:34.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:07:34.592 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 85 up_thru 85 down_at 61 last_clean_interval [10,58) [v2:127.0.0.1:6818/1685115119,v1:127.0.0.1:6819/1685115119] [v2:127.0.0.1:6820/1685115119,v1:127.0.0.1:6821/1685115119] exists,up 427ee48c-66f3-493b-b6c3-1c333687da39 2024-09-01T03:07:34.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:07:34.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:07:34.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:07:34.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-01T03:07:34.991 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:34.990+0000 7f21c761e640 -1 Falling back to public interface 2024-09-01T03:07:38.029 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:07:38.027+0000 7f21c761e640 -1 osd.1 87 log_to_monitors true 2024-09-01T03:07:54.596 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-01T03:07:54.596 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T03:07:54.596 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T03:07:54.596 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:07:54.596 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:07:54.597 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:07:54.597 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T03:07:54.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-01T03:07:54.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok dump_ops_in_flight 2024-09-01T03:07:54.683 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:07:54.683 INFO:tasks.workunit.client.0.smithi079.stdout: "ops": [], 2024-09-01T03:07:54.683 INFO:tasks.workunit.client.0.smithi079.stdout: "num_ops": 0 2024-09-01T03:07:54.683 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:07:54.691 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-09-01T03:07:54.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-01T03:07:54.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:55.072 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:55.072 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688500+0000 36'51 92:155 [2,0,1] 2 [2,0,1] 2 36'51 2024-09-01T03:07:28.623114+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:59:32.080294+0000 3 0 2024-09-01T03:07:55.072 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692719+0000 42'51 92:198 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:46:01.712748+0000 2 0 2024-09-01T03:07:55.072 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688869+0000 38'51 92:145 [2,1,0] 2 [2,1,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:16:00.186283+0000 4 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:43.954644+0000 41'50 92:183 [1,0,2] 1 [1,0,2] 1 41'50 2024-09-01T03:07:43.954564+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:38:18.768310+0000 3 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:39.692296+0000 34'50 92:142 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:39.108302+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:06:32.163364+0000 3 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:07:39.847731+0000 41'51 92:158 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:36.123925+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:14:50.352742+0000 4 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1449 0 0 16 36 16 active+clean 2024-09-01T03:07:42.021593+0000 42'52 92:174 [1,2,0] 1 [1,2,0] 1 42'52 2024-09-01T03:07:42.021533+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T11:14:58.083770+0000 3 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:07:39.687307+0000 34'50 92:160 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:07:28.705116+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:56:28.801419+0000 2 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 0 0 0 2415 0 0 17 36 17 active+clean 2024-09-01T03:07:44.918895+0000 87'53 92:169 [1,0,2] 1 [1,0,2] 1 87'53 2024-09-01T03:07:44.918829+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T09:54:33.269208+0000 5 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 2898 0 0 16 36 16 active+clean 2024-09-01T03:07:39.688163+0000 41'52 92:167 [2,0,1] 2 [2,0,1] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:38:37.177831+0000 6 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:07:42.997485+0000 34'50 92:146 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:07:42.997393+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:32:38.574839+0000 2 0 2024-09-01T03:07:55.073 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:07:39.690215+0000 43'51 92:145 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:02:45.511694+0000 4 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:07:39.694662+0000 38'50 92:154 [0,1,2] 0 [0,1,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:43:54.095249+0000 1 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692360+0000 41'51 92:143 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:04:34.366914+0000 2 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 966 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688238+0000 41'51 92:151 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:09:49.873054+0000 0 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:07:40.049746+0000 32'49 92:135 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:07:40.049687+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:07:05.722371+0000 2 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:39.693055+0000 42'50 92:145 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:52:39.098225+0000 2 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688023+0000 41'51 92:152 [2,1,0] 2 [2,1,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:17:39.639340+0000 1 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 1 1 0 0 1603 0 0 16 36 16 active+recovery_wait+degraded 2024-09-01T03:07:39.693329+0000 80'52 92:188 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 2 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 1932 0 0 14 36 14 active+clean 2024-09-01T03:07:39.690621+0000 36'50 92:143 [2,0,1] 2 [2,0,1] 2 36'50 2024-09-01T03:07:27.669943+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:53:53.174213+0000 4 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692437+0000 42'51 92:166 [0,2,1] 0 [0,2,1] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:18:04.206359+0000 2 0 2024-09-01T03:07:55.074 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 1 0 0 1223 0 0 14 36 14 active+recovering+degraded 2024-09-01T03:07:39.848092+0000 34'50 92:133 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 no scrub is scheduled 0 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:07:39.688872+0000 34'50 92:164 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:20:03.555799+0000 1 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:41.030061+0000 42'51 92:172 [1,0,2] 1 [1,0,2] 1 42'51 2024-09-01T03:07:41.029999+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:48:58.848615+0000 3 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.690905+0000 42'51 92:160 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:26:16.859297+0000 1 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:45.869268+0000 36'50 92:120 [1,0,2] 1 [1,0,2] 1 36'50 2024-09-01T03:07:45.869210+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T08:53:45.259155+0000 3 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692900+0000 36'51 92:143 [1,0,2] 1 [1,0,2] 1 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:34:09.811540+0000 4 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 483 0 0 13 36 13 active+clean 2024-09-01T03:07:39.694169+0000 32'49 92:131 [0,1,2] 0 [0,1,2] 0 32'49 2024-09-01T03:07:27.181484+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:11:35.959817+0000 1 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692931+0000 42'51 92:165 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:18.542213+0000 2 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:07:39.688857+0000 32'49 92:145 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:07:27.735324+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:23:28.829297+0000 2 0 2024-09-01T03:07:55.075 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:39.696191+0000 41'51 92:157 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:40.110498+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:36:11.401444+0000 4 0 2024-09-01T03:07:55.076 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 1 1 0 0 2415 0 0 15 36 15 active+recovery_wait+degraded 2024-09-01T03:07:39.692669+0000 42'51 92:173 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 4 0 2024-09-01T03:07:55.076 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:55.076 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:55.076 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:55.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-01T03:07:55.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-01T03:07:55.193 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:07:55.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-01T03:07:55.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-01T03:07:55.315 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:07:55.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-01T03:07:55.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-01T03:07:55.437 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:07:55.445 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-09-01T03:07:55.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-01T03:07:55.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.1f 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688500+0000 36'51 92:155 [2,0,1] 2 [2,0,1] 2 36'51 2024-09-01T03:07:28.623114+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:59:32.080294+0000 3 0 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.1e 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692719+0000 42'51 92:198 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:23.511513+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:46:01.712748+0000 2 0 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.1d 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688869+0000 38'51 92:145 [2,1,0] 2 [2,1,0] 2 38'51 2024-09-01T03:05:23.045105+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:16:00.186283+0000 4 0 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.b 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:43.954644+0000 41'50 92:183 [1,0,2] 1 [1,0,2] 1 41'50 2024-09-01T03:07:43.954564+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:38:18.768310+0000 3 0 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.a 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:39.692296+0000 34'50 92:142 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:39.108302+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:06:32.163364+0000 3 0 2024-09-01T03:07:55.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.9 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:07:39.847731+0000 41'51 92:158 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:36.123925+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:14:50.352742+0000 4 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.8 3 0 0 0 0 1449 0 0 16 36 16 active+clean 2024-09-01T03:07:42.021593+0000 42'52 92:174 [1,2,0] 1 [1,2,0] 1 42'52 2024-09-01T03:07:42.021533+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T11:14:58.083770+0000 3 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.7 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:07:39.687307+0000 34'50 92:160 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:07:28.705116+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:56:28.801419+0000 2 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.6 5 0 0 0 0 2415 0 0 17 36 17 active+clean 2024-09-01T03:07:44.918895+0000 87'53 92:169 [1,0,2] 1 [1,0,2] 1 87'53 2024-09-01T03:07:44.918829+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T09:54:33.269208+0000 5 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.5 6 0 0 0 0 2898 0 0 16 36 16 active+clean 2024-09-01T03:07:39.688163+0000 41'52 92:167 [2,0,1] 2 [2,0,1] 2 41'52 2024-09-01T03:05:28.167637+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T03:38:37.177831+0000 6 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.4 2 0 0 0 0 966 0 0 14 36 14 active+clean 2024-09-01T03:07:42.997485+0000 34'50 92:146 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:07:42.997393+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:32:38.574839+0000 2 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.3 5 0 0 0 0 2415 0 0 15 36 15 active+clean 2024-09-01T03:07:39.690215+0000 43'51 92:145 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:18.298323+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:02:45.511694+0000 4 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.2 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:07:39.694662+0000 38'50 92:154 [0,1,2] 0 [0,1,2] 0 38'50 2024-09-01T03:05:28.519118+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:43:54.095249+0000 1 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.c 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692360+0000 41'51 92:143 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:23.327329+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:04:34.366914+0000 2 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.1 2 0 0 0 0 966 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688238+0000 41'51 92:151 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:21.164294+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:09:49.873054+0000 0 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.1c 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:07:40.049746+0000 32'49 92:135 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:07:40.049687+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T05:07:05.722371+0000 2 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:39.693055+0000 42'50 92:145 [1,0,2] 1 [1,0,2] 1 32'49 2024-09-01T03:05:16.301963+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T14:52:39.098225+0000 2 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.d 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.688023+0000 41'51 92:152 [2,1,0] 2 [2,1,0] 2 32'49 2024-09-01T03:05:22.112451+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:17:39.639340+0000 1 0 2024-09-01T03:07:55.820 INFO:tasks.workunit.client.0.smithi079.stdout:1.e 4 1 1 0 0 1603 0 0 16 36 16 active+recovery_wait+degraded 2024-09-01T03:07:39.693329+0000 80'52 92:188 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:31.138489+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 2 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.f 4 0 0 0 0 1932 0 0 14 36 14 active+clean 2024-09-01T03:07:39.690621+0000 36'50 92:143 [2,0,1] 2 [2,0,1] 2 36'50 2024-09-01T03:07:27.669943+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:53:53.174213+0000 4 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.10 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692437+0000 42'51 92:166 [0,2,1] 0 [0,2,1] 0 34'50 2024-09-01T03:05:27.345888+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T08:18:04.206359+0000 2 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.11 3 0 1 0 0 1223 0 0 14 36 14 active+recovering+degraded 2024-09-01T03:07:39.848092+0000 34'50 92:133 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.067899+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 no scrub is scheduled 0 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.12 1 0 0 0 0 483 0 0 14 36 14 active+clean 2024-09-01T03:07:39.688872+0000 34'50 92:164 [0,1,2] 0 [0,1,2] 0 34'50 2024-09-01T03:05:27.563147+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T10:20:03.555799+0000 1 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.13 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:41.030061+0000 42'51 92:172 [1,0,2] 1 [1,0,2] 1 42'51 2024-09-01T03:07:41.029999+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:48:58.848615+0000 3 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.14 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.690905+0000 42'51 92:160 [2,0,1] 2 [2,0,1] 2 32'49 2024-09-01T03:05:22.429192+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T12:26:16.859297+0000 1 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.15 3 0 0 0 0 1449 0 0 14 36 14 active+clean 2024-09-01T03:07:45.869268+0000 36'50 92:120 [1,0,2] 1 [1,0,2] 1 36'50 2024-09-01T03:07:45.869210+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T08:53:45.259155+0000 3 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.16 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692900+0000 36'51 92:143 [1,0,2] 1 [1,0,2] 1 36'51 2024-09-01T03:05:32.162730+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T04:34:09.811540+0000 4 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.17 1 0 0 0 0 483 0 0 13 36 13 active+clean 2024-09-01T03:07:39.694169+0000 32'49 92:131 [0,1,2] 0 [0,1,2] 0 32'49 2024-09-01T03:07:27.181484+0000 0'0 2024-09-01T03:04:54.895494+0000 0 0 periodic scrub scheduled @ 2024-09-02T03:11:35.959817+0000 1 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.18 3 0 0 0 0 1449 0 0 15 36 15 active+clean 2024-09-01T03:07:39.692931+0000 42'51 92:165 [1,2,0] 1 [1,2,0] 1 34'50 2024-09-01T03:05:21.364021+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T11:40:18.542213+0000 2 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.19 2 0 0 0 0 966 0 0 13 36 13 active+clean 2024-09-01T03:07:39.688857+0000 32'49 92:145 [1,2,0] 1 [1,2,0] 1 32'49 2024-09-01T03:07:27.735324+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T09:23:28.829297+0000 2 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.1a 4 0 0 0 0 1932 0 0 15 36 15 active+clean 2024-09-01T03:07:39.696191+0000 41'51 92:157 [1,2,0] 1 [1,2,0] 1 41'51 2024-09-01T03:05:40.110498+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 periodic scrub scheduled @ 2024-09-02T07:36:11.401444+0000 4 0 2024-09-01T03:07:55.821 INFO:tasks.workunit.client.0.smithi079.stdout:1.1b 5 1 1 0 0 2415 0 0 15 36 15 active+recovery_wait+degraded 2024-09-01T03:07:39.692669+0000 42'51 92:173 [1,0,2] 1 [1,0,2] 1 34'50 2024-09-01T03:05:26.313835+0000 0'0 2024-09-01T03:04:54.895494+0000 0 1 no scrub is scheduled 4 0 2024-09-01T03:07:55.822 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:07:55.822 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:07:55.822 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:07:55.830 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-01T03:07:55.831 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-01T03:07:55.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:07:55.832 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-01T03:07:55.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:07:55.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-01T03:07:55.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:07:55.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-01T03:07:55.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:07:55.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-01T03:07:55.987 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-01T03:07:55.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:07:55.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-01T03:07:56.022 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-01T03:07:56.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-01T03:07:56.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-01T03:07:56.058 INFO:tasks.workunit.client.0.smithi079.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-01T03:07:56.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-01T03:07:56.060 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-01T03:07:56.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 49'51 (31'3) delete 1:8881e6d8:::existing_3:head by client.4629.0:1 2024-09-01T03:05:36.169855+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 20 _merge_object_divergent_entries: keeping 49'51 (31'3) delete 1:8881e6d8:::existing_3:head by client.4629.0:1 2024-09-01T03:05:36.169855+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 49'51 last_divergent_update: 49'51 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 49'51 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 49'51 (31'3) delete 1:8881e6d8:::existing_3:head by client.4629.0:1 2024-09-01T03:05:36.169855+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-01T03:07:56.083 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.0.log:2024-09-01T03:07:26.126+0000 7f5d8094d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 32'36 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 49'52 (31'2) modify 1:97db2a0f:::existing_2:head by client.4641.0:1 2024-09-01T03:05:36.189995+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 20 _merge_object_divergent_entries: keeping 49'52 (31'2) modify 1:97db2a0f:::existing_2:head by client.4641.0:1 2024-09-01T03:05:36.189995+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 49'52 last_divergent_update: 49'52 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 49'52 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 49'52 (31'2) modify 1:97db2a0f:::existing_2:head by client.4641.0:1 2024-09-01T03:05:36.189995+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.125+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 32'36 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 49'52 (31'1) modify 1:d858c035:::existing_1:head by client.4633.0:1 2024-09-01T03:05:36.181739+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:07:56.084 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 20 _merge_object_divergent_entries: keeping 49'52 (31'1) modify 1:d858c035:::existing_1:head by client.4633.0:1 2024-09-01T03:05:36.181739+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 49'52 last_divergent_update: 49'52 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 49'53 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 49'52 (31'1) modify 1:d858c035:::existing_1:head by client.4633.0:1 2024-09-01T03:05:36.181739+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 32'36 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 49'53 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4639.0:1 2024-09-01T03:05:36.187478+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 20 _merge_object_divergent_entries: keeping 49'53 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4639.0:1 2024-09-01T03:05:36.187478+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 49'53 last_divergent_update: 49'53 2024-09-01T03:07:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 49'53 (31'4) delete 1:70b3bf12:::existing_4:head by client.4636.0:1 2024-09-01T03:05:36.183903+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 20 _merge_object_divergent_entries: keeping 49'53 (31'4) delete 1:70b3bf12:::existing_4:head by client.4636.0:1 2024-09-01T03:05:36.183903+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 49'53 last_divergent_update: 49'53 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: more recent entry found: 80'52 (31'4) modify 1:70b3bf12:::existing_4:head by client.4760.0:1 2024-09-01T03:06:40.777274+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false, already merged 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 49'52 (0'0) modify 1:75ac5091:::newobject_2:head by client.4627.0:1 2024-09-01T03:05:36.166495+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 20 _merge_object_divergent_entries: keeping 49'52 (0'0) modify 1:75ac5091:::newobject_2:head by client.4627.0:1 2024-09-01T03:05:36.166495+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 49'52 last_divergent_update: 49'52 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-01T03:07:56.086 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.158+0000 7f41cedcc640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-01T03:07:56.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 49'51 (31'3) delete 1:8881e6d8:::existing_3:head by client.4629.0:1 2024-09-01T03:05:36.169855+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 20 _merge_object_divergent_entries: keeping 49'51 (31'3) delete 1:8881e6d8:::existing_3:head by client.4629.0:1 2024-09-01T03:05:36.169855+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-01T03:07:56.087 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 49'51 last_divergent_update: 49'51 2024-09-01T03:07:56.095 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-01T03:07:56.095 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:07:56.095 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 49'51 2024-09-01T03:07:56.095 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 49'51 (31'3) delete 1:8881e6d8:::existing_3:head by client.4629.0:1 2024-09-01T03:05:36.169855+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-01T03:07:56.095 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-01T03:07:56.095 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:07:26.701+0000 7f41cfdce640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 32'36 2024-09-01T03:07:56.096 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-01T03:07:56.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:07:56.115 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-01T03:07:56.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:07:56.136 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-09-01T03:07:56.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-01T03:07:56.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.22149.out 2024-09-01T03:07:56.137 INFO:tasks.workunit.client.0.smithi079.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.22149.out': No such file or directory 2024-09-01T03:07:56.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-01T03:07:56.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-01T03:07:56.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-01T03:07:56.616 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-09-01T03:07:56.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-01T03:07:56.628 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:07:56.628 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:07:56.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:07:56.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:07:56.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:08:01.943 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:08:01.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:08:01.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:08:01.949 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:08:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:08:01.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:08:01.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:08:01.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:08:01.952 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:08:01.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:08:01.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:08:01.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:08:01.954 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:08:01.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:08:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:08:01.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T03:08:01.993 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:08:01.993 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:01.993 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:01.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T03:08:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:08:01.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:08:01.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:08:01.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:08:01.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:08:01.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:08:01.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:08:02.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:08:02.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:08:02.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:08:02.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:08:02.002 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:08:02.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:08:02.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:08:02.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T03:08:02.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:08:02.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:02.004 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:02.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T03:08:02.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:08:02.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:08:02.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-01T03:08:02.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:08:02.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:02.007 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:02.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.22149 2024-09-01T03:08:02.009 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:08:02.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-01T03:08:02.010 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-01T03:08:02.011 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-01T03:08:02.012 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-01T03:08:02.480 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:08:02.481 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:08:02.481 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:08:02.481 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:08:02.481 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:02.481 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:02.482 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:08:02.482 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:08:02.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:08:02.683 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:08:02.683 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:08:02.683 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:08:02.683 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:08:02.683 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:08:02.684 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get fsid 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:08:02.767 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:08:02.768 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:08:02.768 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:02.768 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:02.768 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-mon.a.asok 2024-09-01T03:08:02.768 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:08:02.768 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.22149/ceph-mon.a.asok config get mon_host 2024-09-01T03:08:02.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-01T03:08:02.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-01T03:08:02.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:08:02.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:08:02.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:08:02.846 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-01T03:08:02.846 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:08:03.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:08:03.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:08:03.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:08:03.694 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:08:03.695 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:03.695 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:03.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:08:03.695 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:08:03.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:08:03.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-01T03:08:03.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-01T03:08:03.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:08:03.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:08:03.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-01T03:08:03.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:08:03.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:08:03.735 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:08:03.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9f792487-0fd9-486c-914e-5de13226be1c' 2024-09-01T03:08:03.735 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:08:03.756 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCT2tNm+Cz+LBAA8/r/PWaXBrNUHxmCwriRZw== 2024-09-01T03:08:03.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCT2tNm+Cz+LBAA8/r/PWaXBrNUHxmCwriRZw=="}' 2024-09-01T03:08:03.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9f792487-0fd9-486c-914e-5de13226be1c -i td/divergent-priors/0/new.json 2024-09-01T03:08:04.038 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:08:04.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-01T03:08:04.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCT2tNm+Cz+LBAA8/r/PWaXBrNUHxmCwriRZw== --osd-uuid 9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:08:04.086 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:04.085+0000 7f3ef540c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:04.099 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:04.097+0000 7f3ef540c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:04.100 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:04.098+0000 7f3ef540c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:04.100 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:04.099+0000 7f3ef540c640 -1 bdev(0x55a4ca309c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:08:04.100 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:04.099+0000 7f3ef540c640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-01T03:08:09.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-01T03:08:09.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:08:09.618 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:08:09.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:08:09.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:08:10.157 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:08:10.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:08:10.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:08:10.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:08:10.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:08:10.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:08:10.193 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:10.191+0000 7f73fb5c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:10.198 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:10.196+0000 7f73fb5c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:10.199 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:10.197+0000 7f73fb5c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:10.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:08:10.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:08:10.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:08:10.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:08:10.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:08:10.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:10.572 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:08:10.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:08:10.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:10.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:10.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:11.044 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:11.043+0000 7f73fb5c3640 -1 Falling back to public interface 2024-09-01T03:08:11.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:11.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:11.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:08:11.962 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:08:11.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:11.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:12.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:13.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:13.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:13.359 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:08:13.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:08:13.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:13.360 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:13.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:14.177 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:14.175+0000 7f73fb5c3640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:08:14.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:14.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:14.757 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:08:14.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:08:14.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:14.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:15.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:16.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:16.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:16.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:08:16.266 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:08:16.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:16.266 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:16.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:17.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:17.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:17.704 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:08:17.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:08:17.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:17.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:18.235 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:19.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:19.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:19.237 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:08:19.237 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:08:19.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:19.238 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:19.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:20.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:20.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:20.669 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:08:20.670 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:08:20.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:20.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:21.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:22.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:22.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:22.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:08:22.126 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:08:22.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:22.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:22.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:23.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:23.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:23.524 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:08:23.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:08:23.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:23.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:24.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:25.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:25.046 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:25.047 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:08:25.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:08:25.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:25.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:25.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:26.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:26.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:26.688 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:08:26.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:08:26.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:26.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:27.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:28.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:28.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:28.331 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:08:28.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:08:28.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:28.332 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:29.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:30.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:30.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:30.089 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:08:30.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:08:30.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:30.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:31.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:32.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:32.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:32.105 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:08:32.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:08:32.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:32.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:32.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:33.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:33.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:33.967 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:08:33.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:08:33.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:33.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:34.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:35.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:35.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:35.952 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:08:35.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:08:35.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:35.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:36.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:37.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:37.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:37.790 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:08:37.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:08:37.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:37.791 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:08:38.185 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1300663816,v1:127.0.0.1:6803/1300663816] [v2:127.0.0.1:6804/1300663816,v1:127.0.0.1:6805/1300663816] exists,up 9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T03:08:38.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:08:38.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:08:38.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8569ecdc-ab13-4c87-a722-5e1735ac0930 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 8569ecdc-ab13-4c87-a722-5e1735ac0930' 2024-09-01T03:08:38.192 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 8569ecdc-ab13-4c87-a722-5e1735ac0930 2024-09-01T03:08:38.193 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:08:38.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC22tNmi/SWDBAA2w7BMRHb67u4xK/9H6CMcQ== 2024-09-01T03:08:38.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC22tNmi/SWDBAA2w7BMRHb67u4xK/9H6CMcQ=="}' 2024-09-01T03:08:38.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8569ecdc-ab13-4c87-a722-5e1735ac0930 -i td/divergent-priors/1/new.json 2024-09-01T03:08:38.680 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:08:38.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-01T03:08:38.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC22tNmi/SWDBAA2w7BMRHb67u4xK/9H6CMcQ== --osd-uuid 8569ecdc-ab13-4c87-a722-5e1735ac0930 2024-09-01T03:08:38.729 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:38.727+0000 7f7fd0f32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:38.731 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:38.729+0000 7f7fd0f32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:38.732 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:38.730+0000 7f7fd0f32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:38.733 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:38.730+0000 7f7fd0f32640 -1 bdev(0x556fd449dc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-01T03:08:38.733 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:38.730+0000 7f7fd0f32640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-01T03:08:44.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-01T03:08:44.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:08:44.155 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T03:08:44.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T03:08:44.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:08:44.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T03:08:44.712 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:08:44.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:08:44.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:08:44.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:08:44.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:08:44.747 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:44.745+0000 7fd709745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:44.749 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:44.747+0000 7fd709745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:44.750 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:44.748+0000 7fd709745640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:08:45.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T03:08:45.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:08:45.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:08:45.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:08:45.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:08:45.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:45.178 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:08:45.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:08:45.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:45.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:45.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:46.091 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:46.090+0000 7fd709745640 -1 Falling back to public interface 2024-09-01T03:08:46.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:46.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:46.606 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:08:46.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:08:46.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:46.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:47.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:48.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:48.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:48.040 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:08:48.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:08:48.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:48.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:48.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:49.227 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:08:49.226+0000 7fd709745640 -1 osd.1 0 log_to_monitors true 2024-09-01T03:08:49.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:49.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:49.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:08:49.476 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:08:49.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:49.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:50.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:51.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:51.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:51.015 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:08:51.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:08:51.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:51.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:51.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:52.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:52.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:52.422 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:08:52.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:08:52.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:52.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:52.842 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:53.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:53.843 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:53.844 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:08:53.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:08:53.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:53.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:54.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:55.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:55.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:55.276 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:08:55.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:08:55.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:55.276 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:55.673 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:56.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:56.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:56.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:08:56.675 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:08:56.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:56.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:57.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:58.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:58.171 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:08:58.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:58.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:08:58.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:58.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:08:58.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:08:59.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:08:59.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:08:59.572 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:08:59.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:08:59.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:08:59.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:00.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:01.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:01.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:01.085 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:09:01.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:09:01.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:01.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:01.738 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:02.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:02.739 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:02.740 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:09:02.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:09:02.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:02.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:03.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:04.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:04.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:04.708 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:09:04.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:09:04.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:04.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:05.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:06.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:06.504 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:09:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:09:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:06.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:07.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:08.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:08.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:08.339 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:09:08.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:09:08.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:08.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:09.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:10.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:10.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:10.131 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:09:10.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:09:10.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:10.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:10.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:11.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:11.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:11.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:09:11.754 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:09:11.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:11.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:09:12.216 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4070221926,v1:127.0.0.1:6811/4070221926] [v2:127.0.0.1:6812/4070221926,v1:127.0.0.1:6813/4070221926] exists,up 8569ecdc-ab13-4c87-a722-5e1735ac0930 2024-09-01T03:09:12.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:09:12.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:09:12.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:09:12.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:09:12.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:09:12.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:09:12.224 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:09:12.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5268afd7-f8a5-4e5f-970d-389d79a07cb5' 2024-09-01T03:09:12.224 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:09:12.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDY2tNmrT5wDhAAj7yB9ImpedsQpGJ+y2psYw== 2024-09-01T03:09:12.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDY2tNmrT5wDhAAj7yB9ImpedsQpGJ+y2psYw=="}' 2024-09-01T03:09:12.244 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5268afd7-f8a5-4e5f-970d-389d79a07cb5 -i td/divergent-priors/2/new.json 2024-09-01T03:09:12.850 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:09:12.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-01T03:09:12.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDY2tNmrT5wDhAAj7yB9ImpedsQpGJ+y2psYw== --osd-uuid 5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:09:12.902 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:12.900+0000 7f54b5eb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:09:12.903 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:12.902+0000 7f54b5eb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:09:12.904 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:12.903+0000 7f54b5eb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:09:12.905 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:12.903+0000 7f54b5eb2640 -1 bdev(0x55ee7f04dc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-01T03:09:12.905 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:12.903+0000 7f54b5eb2640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-01T03:09:18.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-01T03:09:18.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:09:18.764 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T03:09:18.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T03:09:18.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:09:19.311 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:09:19.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T03:09:19.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:09:19.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:09:19.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:09:19.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:09:19.347 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:19.345+0000 7f2e33c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:09:19.349 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:19.347+0000 7f2e33c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:09:19.350 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:19.348+0000 7f2e33c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:09:19.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T03:09:19.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:09:19.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:09:19.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:09:19.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:09:19.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:19.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:09:19.779 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:09:19.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:19.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:20.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:20.704 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:20.701+0000 7f2e33c15640 -1 Falling back to public interface 2024-09-01T03:09:21.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:21.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:21.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:09:21.224 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:09:21.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:21.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:21.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:22.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:22.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:22.658 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:09:22.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:09:22.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:22.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:23.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:23.520 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:09:23.519+0000 7f2e33c15640 -1 osd.2 0 log_to_monitors true 2024-09-01T03:09:24.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:24.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:24.062 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:09:24.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:09:24.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:24.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:24.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:25.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:25.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:25.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:09:25.556 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:09:25.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:25.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:26.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:27.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:27.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:27.109 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:09:27.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:09:27.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:27.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:27.679 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:28.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:28.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:28.681 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:09:28.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:09:28.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:28.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:29.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:30.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:30.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:30.077 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:09:30.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:09:30.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:30.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:30.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:31.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:31.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:31.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:09:31.614 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:09:31.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:31.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:32.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:33.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:33.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:33.221 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:09:33.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:09:33.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:33.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:33.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:34.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:34.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:34.644 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:09:34.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:09:34.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:34.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:35.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:36.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:36.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:36.084 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:09:36.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:09:36.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:36.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:37.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:38.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:38.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:38.099 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:09:38.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:09:38.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:38.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:38.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:39.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:39.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:39.985 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:09:39.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:09:39.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:39.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:40.755 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:41.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:41.757 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:41.757 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:09:41.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:09:41.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:41.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:42.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:43.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:43.687 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:43.687 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:09:43.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:09:43.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:43.688 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:44.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:45.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:45.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:45.275 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:09:45.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:09:45.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:45.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:46.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:09:47.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:09:47.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:09:47.110 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:09:47.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:09:47.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:09:47.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:09:47.548 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3621364537,v1:127.0.0.1:6819/3621364537] [v2:127.0.0.1:6820/3621364537,v1:127.0.0.1:6821/3621364537] exists,up 5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:09:47.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:09:47.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:09:47.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:09:47.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-01T03:09:48.356 INFO:tasks.workunit.client.0.smithi079.stderr:noout is set 2024-09-01T03:09:48.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-01T03:09:49.109 INFO:tasks.workunit.client.0.smithi079.stderr:noin is set 2024-09-01T03:09:49.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-01T03:09:49.929 INFO:tasks.workunit.client.0.smithi079.stderr:nodown is set 2024-09-01T03:09:49.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-01T03:09:49.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=test 2024-09-01T03:09:49.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-01T03:09:49.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-01T03:09:49.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-01T03:09:49.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-01T03:09:50.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-01T03:09:50.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-01T03:09:51.367 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' already exists 2024-09-01T03:09:51.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T03:09:52.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-01T03:09:52.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-01T03:09:53.179 INFO:tasks.workunit.client.0.smithi079.stderr:set pool 1 allow_ec_overwrites to true 2024-09-01T03:09:53.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:09:53.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:09:53.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:09:53.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:09:53.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:09:53.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:09:53.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:09:53.280 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:09:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:09:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:09:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:09:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:09:53.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:09:53.760 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:09:53.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-01T03:09:53.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-01T03:09:53.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-01T03:09:53.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:09:53.911 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:09:54.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-01T03:09:54.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-01T03:09:54.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672974' 2024-09-01T03:09:54.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:09:54.030 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:09:54.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-01T03:09:54.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-01T03:09:54.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672974 2-64424509454' 2024-09-01T03:09:54.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:09:54.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-01T03:09:54.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:09:54.151 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:09:54.151 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-01T03:09:54.151 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:09:54.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-01T03:09:54.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-01T03:09:54.153 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836499 2024-09-01T03:09:54.153 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:09:54.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-01T03:09:54.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:09:55.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:09:55.623 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:09:56.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-01T03:09:56.081 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:09:56.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-01T03:09:56.082 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:09:56.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:09:56.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-01T03:09:56.083 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:09:56.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-01T03:09:56.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-01T03:09:56.085 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672974 2024-09-01T03:09:56.085 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:09:56.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-01T03:09:56.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:09:56.508 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-01T03:09:56.508 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:09:56.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:09:56.509 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-01T03:09:56.509 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:09:56.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-01T03:09:56.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-01T03:09:56.511 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509454 2024-09-01T03:09:56.511 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:09:56.942 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-01T03:09:56.942 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:09:56.942 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:09:56.943 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:09:57.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:09:57.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:09:57.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:09:57.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:09:57.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:09:57.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:09:57.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:09:57.469 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:09:57.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:09:57.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:09:57.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:09:57.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:09:58.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:09:58.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:09:58.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:09:58.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-01T03:09:58.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-01T03:09:58.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:09:58.396 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:09:58.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:09:58.818 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:09:58.818 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:09:58.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:09:58.818 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:09:58.818 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:09:58.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-01T03:09:58.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-01T03:09:58.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-01T03:09:58.939 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:09:58.939 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:09:59.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-01T03:09:59.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-01T03:09:59.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-09-01T03:09:59.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:09:59.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:09:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-01T03:09:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-01T03:09:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509457' 2024-09-01T03:09:59.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:09:59.179 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-01T03:09:59.179 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:09:59.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:09:59.180 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-01T03:09:59.180 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:09:59.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-01T03:09:59.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-01T03:09:59.182 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836502 2024-09-01T03:09:59.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:09:59.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-01T03:09:59.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:10:00.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:10:00.637 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:10:01.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-01T03:10:01.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:01.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-01T03:10:01.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:01.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:10:01.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-01T03:10:01.059 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:01.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-01T03:10:01.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-01T03:10:01.061 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672977 2024-09-01T03:10:01.061 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:10:01.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-01T03:10:01.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:01.492 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-01T03:10:01.492 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:01.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:10:01.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-01T03:10:01.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:01.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-01T03:10:01.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-01T03:10:01.495 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509457 2024-09-01T03:10:01.495 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:10:01.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-01T03:10:01.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:10:01.890 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:10:01.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:10:01.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:10:01.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:10:01.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:10:01.980 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:10:01.980 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:10:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:10:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:10:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:10:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:10:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:10:02.411 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:10:02.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-01T03:10:02.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-01T03:10:02.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-01T03:10:02.531 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:10:02.531 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:10:02.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-01T03:10:02.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-01T03:10:02.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672979' 2024-09-01T03:10:02.652 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:10:02.652 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:10:02.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-01T03:10:02.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-01T03:10:02.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672979 2-64424509459' 2024-09-01T03:10:02.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:02.773 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-01T03:10:02.773 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:02.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:10:02.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-01T03:10:02.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:02.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-01T03:10:02.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-01T03:10:02.775 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836504 2024-09-01T03:10:02.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:10:03.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-01T03:10:03.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:10:04.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:10:04.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:10:04.638 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-01T03:10:04.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:04.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-01T03:10:04.639 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:04.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:10:04.640 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-01T03:10:04.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:04.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-01T03:10:04.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-01T03:10:04.644 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672979 2024-09-01T03:10:04.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:10:05.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672979 2024-09-01T03:10:05.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:05.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-01T03:10:05.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:05.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:10:05.067 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-01T03:10:05.067 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:05.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-01T03:10:05.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-01T03:10:05.068 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509459 2024-09-01T03:10:05.068 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:10:05.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509459 2024-09-01T03:10:05.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:10:05.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:10:05.456 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:10:05.981 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:10:05.982 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:10:06.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:10:06.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:10:06.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:10:06.362 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:10:06.886 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:10:06.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:10:06.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:10:06.887 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-01T03:10:06.887 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-01T03:10:07.272 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:10:07.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-01T03:10:07.284 INFO:tasks.workunit.client.0.smithi079.stdout:primary and soon to be divergent is 1 2024-09-01T03:10:07.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-01T03:10:07.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:10:07.654 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:10:07.654 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-01T03:09:52.299174+0000 0'0 29:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T10:50:02.521975+0000 0 0 2024-09-01T03:10:07.654 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:10:07.654 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:10:07.654 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-01T03:10:07.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-01T03:10:07.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-01T03:10:07.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-01T03:10:07.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-01T03:10:07.667 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-01T03:10:07.667 INFO:tasks.workunit.client.0.smithi079.stdout:writing initial objects 2024-09-01T03:10:07.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-01T03:10:07.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:07.668 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-01T03:10:07.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:07.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-01T03:10:08.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:08.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-01T03:10:08.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:08.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-01T03:10:08.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:08.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-01T03:10:08.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:08.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-01T03:10:08.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:08.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-01T03:10:09.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:09.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-01T03:10:09.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:09.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-01T03:10:09.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:09.504 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-01T03:10:09.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:09.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-01T03:10:09.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:09.873 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-01T03:10:10.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:10.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-01T03:10:10.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:10.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-01T03:10:10.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:10.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-01T03:10:10.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:10.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-01T03:10:10.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:10.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-01T03:10:11.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:11.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-01T03:10:11.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:11.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-01T03:10:11.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:11.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-01T03:10:11.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:11.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-01T03:10:11.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:11.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-01T03:10:12.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:12.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-01T03:10:12.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:12.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-01T03:10:12.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:12.609 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-01T03:10:12.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:12.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-01T03:10:12.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:12.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-01T03:10:13.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:13.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-01T03:10:13.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:13.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-01T03:10:13.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:13.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-01T03:10:13.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:13.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-01T03:10:14.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:14.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-01T03:10:14.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:14.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-01T03:10:14.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:14.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-01T03:10:14.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:14.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-01T03:10:14.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:14.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-01T03:10:15.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:15.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-01T03:10:15.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:15.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-01T03:10:15.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:15.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-01T03:10:15.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:15.621 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-01T03:10:15.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:15.806 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-01T03:10:16.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:16.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-01T03:10:16.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:16.217 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-01T03:10:16.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:16.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-01T03:10:16.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:16.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-01T03:10:16.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:16.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-01T03:10:16.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:16.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-01T03:10:17.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:17.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-01T03:10:17.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:17.366 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-01T03:10:17.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:17.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-01T03:10:17.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:17.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-01T03:10:18.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:18.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-01T03:10:18.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:18.256 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-01T03:10:18.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:18.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-01T03:10:18.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:18.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-01T03:10:18.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:18.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-01T03:10:19.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:19.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-01T03:10:19.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:19.304 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-01T03:10:19.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:19.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-01T03:10:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:19.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-01T03:10:19.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:19.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-01T03:10:20.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:20.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-01T03:10:20.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:20.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-01T03:10:20.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:20.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-01T03:10:20.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:20.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-01T03:10:20.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:20.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-01T03:10:21.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:21.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-01T03:10:21.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:21.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-01T03:10:21.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:21.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-01T03:10:21.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:21.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-01T03:10:22.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:22.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-01T03:10:22.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:22.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-01T03:10:22.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:22.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-01T03:10:22.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:22.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-01T03:10:22.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:22.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-01T03:10:23.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:23.096 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-01T03:10:23.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:23.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-01T03:10:23.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:23.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-01T03:10:23.608 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:23.608 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-01T03:10:23.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:23.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-01T03:10:24.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:24.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-01T03:10:24.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:24.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-01T03:10:24.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:24.498 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-01T03:10:24.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:24.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-01T03:10:24.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:24.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-01T03:10:25.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:25.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-01T03:10:25.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:25.329 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-01T03:10:25.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:25.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-01T03:10:25.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:25.742 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-01T03:10:25.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:25.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-01T03:10:26.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:26.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-01T03:10:26.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:26.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-01T03:10:26.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:26.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-01T03:10:26.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:26.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-01T03:10:26.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:26.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-01T03:10:27.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:27.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-01T03:10:27.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:27.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-01T03:10:27.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:27.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-01T03:10:27.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:27.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-01T03:10:27.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-01T03:10:27.988 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-01T03:10:28.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:10:28.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-01T03:10:28.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:10:28.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:10:28.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:10:28.199 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:10:28.200 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:10:28.200 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:10:28.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:10:28.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:10:28.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:10:28.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:10:28.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:10:28.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:10:28.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:10:28.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:10:28.255 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:10:28.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:10:28.677 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:10:28.677 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:10:28.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:10:28.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:10:28.677 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:10:28.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-01T03:10:28.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-01T03:10:28.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-01T03:10:28.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:10:28.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:10:28.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-01T03:10:28.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-01T03:10:28.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672987' 2024-09-01T03:10:28.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:10:28.917 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:10:29.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-01T03:10:29.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-01T03:10:29.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672987 2-64424509467' 2024-09-01T03:10:29.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:29.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-01T03:10:29.035 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:29.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:10:29.036 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-01T03:10:29.036 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:29.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-01T03:10:29.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-01T03:10:29.038 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 21474836512 2024-09-01T03:10:29.038 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:10:29.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836512 2024-09-01T03:10:29.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:10:30.479 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:10:30.479 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:10:30.921 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-01T03:10:30.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:30.922 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-01T03:10:30.922 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:30.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:10:30.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-01T03:10:30.924 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:30.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-01T03:10:30.924 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-01T03:10:30.925 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 42949672987 2024-09-01T03:10:30.925 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:10:31.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-01T03:10:31.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:10:31.347 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-01T03:10:31.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:10:31.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:10:31.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-01T03:10:31.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:10:31.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-01T03:10:31.350 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 64424509467 2024-09-01T03:10:31.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-01T03:10:31.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:10:31.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-01T03:10:31.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:10:31.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:10:31.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:10:32.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:10:32.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:10:32.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:10:32.731 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:10:32.731 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:10:32.731 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:10:33.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:10:33.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:10:33.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:10:33.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-01T03:10:33.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-01T03:10:33.271 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-01T03:10:33.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-01T03:10:33.272 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-01T03:10:33.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-01T03:10:33.737 INFO:tasks.workunit.client.0.smithi079.stdout:blackholing osds 0 2 2024-09-01T03:10:33.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-01T03:10:33.737 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:10:34.108 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:10:34.108 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-01T03:09:52.299174+0000 29'100 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T10:50:02.521975+0000 0 0 2024-09-01T03:10:34.108 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:10:34.108 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:10:34.108 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:10:34.119 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:34.119 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-01T03:10:34.119 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T03:10:34.119 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T03:10:34.120 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:10:34.120 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:10:34.120 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:10:34.120 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T03:10:34.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-01T03:10:34.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-01T03:10:34.196 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:10:34.196 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T03:10:34.196 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:10:34.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:34.203 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-01T03:10:34.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T03:10:34.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T03:10:34.205 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:10:34.205 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:10:34.205 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:10:34.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T03:10:34.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-01T03:10:34.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-01T03:10:34.283 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:10:34.283 INFO:tasks.workunit.client.0.smithi079.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-01T03:10:34.283 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:10:34.290 INFO:tasks.workunit.client.0.smithi079.stdout:writing divergent object 2024-09-01T03:10:34.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-01T03:10:34.290 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-01T03:10:34.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-01T03:10:35.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-01T03:10:35.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-01T03:10:35.511 INFO:tasks.workunit.client.0.smithi079.stdout:created pool test snap snap1 2024-09-01T03:10:35.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-01T03:10:35.514 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-01T03:10:35.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-01T03:10:45.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-01T03:10:45.520 INFO:tasks.workunit.client.0.smithi079.stderr:rados: no process found 2024-09-01T03:10:45.520 INFO:tasks.workunit.client.0.smithi079.stdout:killing all the osds 2024-09-01T03:10:45.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-01T03:10:45.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:10:45.902 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:10:45.902 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 active+clean 2024-09-01T03:09:52.299174+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:51:44.807989+0000 0 0 2024-09-01T03:10:45.902 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:10:45.902 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:10:45.902 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:10:45.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-01T03:10:45.914 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:10:45.914 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:10:45.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:10:45.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:10:45.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:10:46.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:10:46.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-01T03:10:46.020 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-01T03:10:46.524 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:10:46.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-01T03:10:46.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-01T03:10:47.045 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T03:10:47.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-01T03:10:47.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-01T03:10:47.552 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T03:10:47.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:47.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-01T03:10:48.078 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already out. 2024-09-01T03:10:48.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:48.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-01T03:10:48.599 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already out. 2024-09-01T03:10:48.610 INFO:tasks.workunit.client.0.smithi079.stdout:bringing up non_divergent 0 2 2024-09-01T03:10:48.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-01T03:10:48.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:10:48.979 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:10:48.979 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 stale+active+clean 2024-09-01T03:09:52.299174+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T06:51:44.807989+0000 0 0 2024-09-01T03:10:48.979 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:10:48.979 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:10:48.979 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:10:48.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:10:48.994 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:10:48.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-01T03:10:48.996 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:10:48.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:10:49.027 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:49.026+0000 7fc28dd89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:10:49.032 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:49.028+0000 7fc28dd89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:10:49.032 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:49.029+0000 7fc28dd89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:10:49.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:10:49.422 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:10:49.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:49.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:10:49.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:10:49.874 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:49.873+0000 7fc28dd89640 -1 Falling back to public interface 2024-09-01T03:10:50.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:10:50.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:50.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:10:50.809 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:10:50.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:50.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:10:51.195 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:10:52.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:10:52.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:52.197 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:10:52.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:10:52.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:52.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:10:52.608 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:10:53.268 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:53.266+0000 7fc28dd89640 -1 osd.0 29 log_to_monitors true 2024-09-01T03:10:53.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:10:53.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:53.610 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:10:53.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:10:53.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:53.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:10:54.065 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1272317485,v1:127.0.0.1:6803/1272317485] [v2:127.0.0.1:6804/1272317485,v1:127.0.0.1:6805/1272317485] exists,up 9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:10:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:10:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:10:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:10:54.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:10:54.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:10:54.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:10:54.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T03:10:54.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T03:10:54.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:10:54.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:10:54.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:10:54.072 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:10:54.072 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:10:54.072 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:10:54.072 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:10:54.072 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:10:54.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:10:54.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:10:54.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:10:54.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:10:54.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:10:54.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:10:54.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-01T03:10:54.075 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:10:54.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:10:54.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:10:54.077 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-01T03:10:54.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:10:54.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:10:54.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:10:54.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:10:54.103 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:54.101+0000 7f2c15422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:10:54.105 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:54.103+0000 7f2c15422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:10:54.106 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:54.104+0000 7f2c15422640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:54.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:10:54.501 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:10:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:54.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:10:54.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:10:55.203 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:55.201+0000 7f2c15422640 -1 Falling back to public interface 2024-09-01T03:10:55.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:10:55.889 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:55.889 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:10:55.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:10:55.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:55.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:10:56.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:10:57.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:10:57.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:57.294 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:10:57.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:10:57.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:57.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:10:57.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:10:58.130 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:10:58.128+0000 7f2c15422640 -1 osd.2 29 log_to_monitors true 2024-09-01T03:10:58.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:10:58.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:10:58.685 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:10:58.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:10:58.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:10:58.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:10:59.111 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/841918556,v1:127.0.0.1:6811/841918556] [v2:127.0.0.1:6812/841918556,v1:127.0.0.1:6813/841918556] exists,up 5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:10:59.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:10:59.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:10:59.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:10:59.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:59.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-01T03:10:59.643 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already in. 2024-09-01T03:10:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:10:59.654 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-01T03:11:00.589 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already in. 2024-09-01T03:11:00.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-01T03:11:05.602 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-01T03:11:05.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-01T03:11:05.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-01T03:11:05.604 INFO:tasks.workunit.client.0.smithi079.stdout:writing non-divergent object existing_4 2024-09-01T03:11:05.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:11:05.986 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:11:05.986 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-01T03:11:00.780491+0000 29'100 43:129 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T13:34:40.877351+0000 0 0 2024-09-01T03:11:05.987 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:11:05.987 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:11:05.987 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:11:05.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-01T03:11:06.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:11:06.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-01T03:11:06.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:11:06.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:11:06.095 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:11:06.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:11:06.095 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:11:06.095 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:11:06.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:11:06.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:11:06.096 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:11:06.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:11:06.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:11:06.149 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:11:06.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:11:06.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:11:06.150 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:11:06.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:11:06.535 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:11:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:11:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:11:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:11:06.536 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:11:06.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789957 2024-09-01T03:11:06.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789957 2024-09-01T03:11:06.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957' 2024-09-01T03:11:06.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:11:06.657 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:11:06.745 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T03:11:06.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T03:11:06.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T03:11:06.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T03:11:06.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:11:06.747 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:11:06.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724548 2024-09-01T03:11:06.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724548 2024-09-01T03:11:06.868 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-09-01T03:11:06.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:11:06.869 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-158913789957 2024-09-01T03:11:06.869 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:11:06.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:11:06.869 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-158913789957 2024-09-01T03:11:06.870 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:11:06.870 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789957 2024-09-01T03:11:06.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-09-01T03:11:06.871 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 158913789957 2024-09-01T03:11:06.871 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:11:07.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789957 2024-09-01T03:11:07.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:11:08.300 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:11:08.301 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:11:08.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789957 -lt 158913789957 2024-09-01T03:11:08.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:11:08.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-167503724548 2024-09-01T03:11:08.733 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:11:08.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:11:08.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-167503724548 2024-09-01T03:11:08.734 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:11:08.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724548 2024-09-01T03:11:08.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-09-01T03:11:08.735 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 167503724548 2024-09-01T03:11:08.736 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:11:09.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724548 -lt 167503724548 2024-09-01T03:11:09.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:11:09.177 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:09.178 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:09.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:09.712 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:10.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:10.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:10.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:10.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:10.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:10.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T03:11:10.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T03:11:10.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T03:11:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:10.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:10.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:11.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:11.112 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:11.113 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:11.113 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:11.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:11.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:11.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:11.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:11.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:11.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:11.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:11.637 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:11.637 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T03:11:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:12.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T03:11:12.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:12.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:12.408 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:12.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:12.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:12.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:12.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:12.409 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:12.789 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:12.789 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:12.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:12.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:13.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:13.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:13.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:13.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:13.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:13.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:13.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:13.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:13.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:13.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:13.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:13.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T03:11:13.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:13.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:14.303 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:14.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:14.684 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:14.684 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:14.684 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:15.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:15.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:15.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:15.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:15.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:15.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:15.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:15.216 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:15.216 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T03:11:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:15.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T03:11:16.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:16.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:16.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:16.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:16.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:16.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:16.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:16.545 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:16.927 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:16.927 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:16.927 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:16.927 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:17.457 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:17.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:18.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:18.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:18.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T03:11:18.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:18.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T03:11:19.659 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:19.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:19.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:19.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:19.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:19.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:19.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:19.661 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:20.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:20.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:20.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:20.049 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:20.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:21.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:21.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:21.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T03:11:21.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:21.122 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T03:11:24.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:24.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:24.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:24.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:24.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:24.324 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:24.325 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:24.325 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:24.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:24.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:24.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:24.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:25.278 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:25.279 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:25.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:25.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:25.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T03:11:25.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:25.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T03:11:32.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:32.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:32.230 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:32.230 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:32.230 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:32.230 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:32.230 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:32.231 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:32.620 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:32.620 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:32.621 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:32.621 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:33.142 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:33.143 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:33.143 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:33.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:33.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:33.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T03:11:33.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:33.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T03:11:40.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:40.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:40.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:40.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:40.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:40.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:40.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:40.967 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:41.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:41.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:41.355 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:41.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:41.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:41.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:41.882 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:42.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:42.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:42.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T03:11:42.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "7715cda0-390e-4300-9c1d-4cd9d4cc77c1", 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T03:11:42.938155+0000", 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T03:11:42.940 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:11:42.941 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-01T03:11:42.942 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:11:42.943 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.944 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:08:02.038114Z", 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:08:02.038114Z", 2024-09-01T03:11:42.945 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T03:11:42.946 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T03:11:42.947 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T03:11:42.948 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 43, 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:08:02.942841+0000", 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:11:00.385430+0000", 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T03:10:58.327460+0000", 2024-09-01T03:11:42.949 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T03:11:00.067410+0000", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 10, 2024-09-01T03:11:42.950 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:11:42.951 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T03:09:51.079565+0000", 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 16389, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 1, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T03:11:42.952 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T03:11:42.953 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "30", 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "pool", 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 1, 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 30, 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [ 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "snapid": 1, 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "stamp": "2024-09-01T03:10:35.329885+0000", 2024-09-01T03:11:42.954 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "snap1" 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T03:11:42.955 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "myprofile", 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T03:11:42.956 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 8192, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {} 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.957 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "9f792487-0fd9-486c-914e-5de13226be1c", 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 5, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 37, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 42, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 31, 2024-09-01T03:11:42.958 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.959 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.960 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.961 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T03:11:42.962 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 1272317485 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/1272317485", 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/1272317485", 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1272317485", 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1272317485", 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.963 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "8569ecdc-ab13-4c87-a722-5e1735ac0930", 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 25, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:11:42.964 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.965 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.966 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:11:42.967 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.968 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/4070221926", 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/4070221926", 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4070221926", 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4070221926", 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.969 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "5268afd7-f8a5-4e5f-970d-389d79a07cb5", 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 15, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 29, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 39, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 33, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:11:42.970 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.971 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.972 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.973 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 841918556 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.974 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/841918556", 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/841918556", 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/841918556", 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/841918556", 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T03:11:42.975 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:10:46.348013+0000", 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:08:36.398738+0000", 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:10:46.890767+0000", 2024-09-01T03:11:42.976 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:09:11.716841+0000", 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:10:47.384243+0000", 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:11:42.977 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:09:46.990184+0000", 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T03:11:42.978 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "myprofile": { 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-device-class": "", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-failure-domain": "osd", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-num-failure-domains": "0", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-osds-per-failure-domain": "0", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-root": "default", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-01T03:11:42.979 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "1", 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van", 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "w": "8" 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T03:11:42.980 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/1272317485,v1:127.0.0.1:6805/1272317485]", 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:11:42.981 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:11:42.982 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:08:08.986284Z", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:11:42.983 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/1272317485,v1:127.0.0.1:6803/1272317485]", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1272317485,v1:127.0.0.1:6809/1272317485]", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1272317485,v1:127.0.0.1:6807/1272317485]", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:11:42.984 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/4070221926,v1:127.0.0.1:6813/4070221926]", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:11:42.985 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T03:11:42.986 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:08:43.518248Z", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:11:42.987 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/4070221926,v1:127.0.0.1:6811/4070221926]", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4070221926,v1:127.0.0.1:6817/4070221926]", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4070221926,v1:127.0.0.1:6815/4070221926]", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:11:42.988 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/841918556,v1:127.0.0.1:6813/841918556]", 2024-09-01T03:11:42.989 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:11:42.990 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:09:18.120448Z", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:11:42.991 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/841918556,v1:127.0.0.1:6811/841918556]", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/841918556,v1:127.0.0.1:6817/841918556]", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/841918556,v1:127.0.0.1:6815/841918556]", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:11:42.992 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.993 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 0, 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [] 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 37 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 39 2024-09-01T03:11:42.994 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 43, 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:11:42.995 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T03:11:42.996 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.997 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.998 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T03:11:42.999 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:11:43.000 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:11:43.001 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.002 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.003 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:11:43.004 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.005 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.006 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.007 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 1, 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "test", 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_chooseleaf_tries", 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 5 2024-09-01T03:11:43.008 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_choose_tries", 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 100 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_indep", 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:11:43.009 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.010 INFO:tasks.workunit.client.0.smithi079.stderr:report 4060889960 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T03:11:43.011 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T03:11:43.012 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T03:08:02:918133+0000", 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T03:11:43.013 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T03:11:43.014 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:11:43.015 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48454, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:11:43.016 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:11:43.017 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:11:43.018 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:11:43.019 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:11:43.020 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83696, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2164, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489104, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036842496, 2024-09-01T03:11:43.021 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2215936, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1550701, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:11:43.022 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T03:11:43.023 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 83696, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2164, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 10, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 81461, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314489104, 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:11:43.024 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322036842496, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2215936, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1550701, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 10390, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 83416938 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:11:43.025 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:11:43.026 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48454, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:11:43.027 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:11:43.028 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:11:43.029 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1241088, 2024-09-01T03:11:43.030 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1241088, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.031 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 158913789964 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672990 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 167503724555 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.032 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2147483647, 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.033 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:11:43.034 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:10:49.666290+0000", 2024-09-01T03:11:43.035 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 99, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 402, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 551, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:11:43.036 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:11:43.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T03:11:43.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:11:43.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-01T03:11:43.038 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:11:43.038 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:11:43.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:11:43.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:11:43.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:11:43.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:11:43.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-01T03:11:43.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:11:43.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:11:43.057 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-01T03:11:43.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:11:43.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-01T03:11:43.058 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-01T03:11:44.533 INFO:tasks.workunit.client.0.smithi079.stderr:PG '1.0' not found 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:11:44.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-01T03:11:44.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:11:44.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:11:44.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:11:44.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:11:44.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:11:44.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:11:44.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:11:44.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-01T03:11:44.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:11:44.812 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:11:44.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:11:44.813 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-01T03:11:44.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:11:44.813 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:11:44.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:11:44.814 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:11:44.847 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:44.845+0000 7f3d1a464640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:11:44.849 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:44.847+0000 7f3d1a464640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:11:44.850 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:44.849+0000 7f3d1a464640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:11:45.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:11:45.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:11:45.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:11:45.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:11:45.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:11:45.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:11:45.246 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:11:45.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:11:45.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:11:45.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:11:45.695 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1272317485,v1:127.0.0.1:6803/1272317485] [v2:127.0.0.1:6804/1272317485,v1:127.0.0.1:6805/1272317485] exists,up 9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:11:45.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:11:45.697 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:11:45.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:11:45.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-01T03:11:45.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:11:45.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:11:45.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-01T03:11:45.802 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:11:45.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-01T03:11:45.803 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-01T03:11:46.205 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:46.204+0000 7f3d1a464640 -1 Falling back to public interface 2024-09-01T03:11:47.128 INFO:tasks.workunit.client.0.smithi079.stderr:PG '1.0' not found 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:11:47.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:11:47.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:11:47.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-01T03:11:47.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:11:47.400 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:11:47.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:11:47.401 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-01T03:11:47.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:11:47.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:11:47.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:11:47.401 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:11:47.436 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:47.434+0000 7fba22c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:11:47.438 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:47.436+0000 7fba22c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:11:47.439 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:47.437+0000 7fba22c18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:11:47.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:11:47.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:11:47.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:11:47.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:11:47.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:11:47.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:11:47.826 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:11:47.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:11:47.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:11:47.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:11:48.252 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/841918556,v1:127.0.0.1:6811/841918556] [v2:127.0.0.1:6812/841918556,v1:127.0.0.1:6813/841918556] exists,up 5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-01T03:11:48.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-01T03:11:48.296 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:48.295+0000 7fba22c18640 -1 Falling back to public interface 2024-09-01T03:11:49.003 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:49.001+0000 7f3d1a464640 -1 osd.0 43 log_to_monitors true 2024-09-01T03:11:50.709 INFO:tasks.workunit.client.0.smithi079.stderr:PG '1.0' not found 2024-09-01T03:11:50.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-01T03:11:50.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-01T03:11:50.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-01T03:11:50.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:11:50.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:11:50.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:11:50.979 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-01T03:11:50.979 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:11:50.980 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:11:50.980 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:11:50.980 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:11:50.980 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:11:50.993 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:11:50.992+0000 7fba22c18640 -1 osd.2 43 log_to_monitors true 2024-09-01T03:11:51.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-01T03:11:51.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:11:51.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:11:51.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:11:51.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:11:51.042 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:11:51.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:11:51.474 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:11:51.474 INFO:tasks.workunit.client.0.smithi079.stderr:2' 2024-09-01T03:11:51.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:11:51.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:11:51.474 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:11:51.595 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-01T03:11:51.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-01T03:11:51.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619' 2024-09-01T03:11:51.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:11:51.596 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:11:51.685 INFO:tasks.workunit.client.0.smithi079.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-01T03:11:51.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-01T03:11:51.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-01T03:11:51.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-01T03:11:51.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:11:51.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:11:51.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-01T03:11:51.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-01T03:11:51.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-09-01T03:11:51.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:11:51.809 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-197568495619 2024-09-01T03:11:51.809 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:11:51.809 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:11:51.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-197568495619 2024-09-01T03:11:51.810 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:11:51.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-01T03:11:51.811 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-09-01T03:11:51.812 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 197568495619 2024-09-01T03:11:51.812 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:11:52.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-01T03:11:52.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:11:52.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-214748364803 2024-09-01T03:11:52.297 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:11:52.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:11:52.298 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-214748364803 2024-09-01T03:11:52.298 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:11:52.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-01T03:11:52.299 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 214748364803 2024-09-01T03:11:52.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-01T03:11:52.299 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:11:52.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 214748364803 2024-09-01T03:11:52.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:11:53.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:11:53.749 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:11:54.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-01T03:11:54.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:11:54.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:54.145 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:54.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:11:54.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:54.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:54.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:54.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:54.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:54.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:54.667 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:55.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:55.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:55.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:55.076 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:55.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:55.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-01T03:11:55.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-01T03:11:55.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-01T03:11:55.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:55.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-01T03:11:55.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:55.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:55.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:55.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:55.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:55.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:55.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:55.703 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:56.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:56.108 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:56.108 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:56.108 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:56.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:56.635 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:57.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:57.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:57.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-01T03:11:57.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:57.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-01T03:11:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:57.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:57.404 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:57.404 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:57.404 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:57.405 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:57.405 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:57.405 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:57.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:57.785 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:57.785 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:57.785 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:11:58.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:11:58.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:11:58.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:11:58.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:11:58.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:11:58.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:11:58.364 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:11:58.364 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:11:58.364 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:11:58.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:11:58.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:11:58.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-01T03:11:58.905 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:11:58.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-01T03:11:59.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:11:59.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:11:59.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:11:59.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:11:59.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:11:59.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:11:59.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:11:59.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:11:59.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:11:59.685 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:11:59.685 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:11:59.685 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:12:00.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:12:00.255 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:12:00.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:12:00.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:12:00.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-01T03:12:00.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:12:00.810 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-01T03:12:01.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:12:01.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:12:01.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:12:01.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:12:01.611 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:12:01.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:12:01.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:12:01.612 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:12:01.995 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:12:01.995 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:12:01.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:12:01.996 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:12:02.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:12:02.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:12:02.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:12:02.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:12:02.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:12:02.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:12:02.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:12:02.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:12:02.515 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:12:03.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:12:03.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:12:03.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-01T03:12:03.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:12:03.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-01T03:12:04.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:12:04.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:12:04.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:12:04.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:12:04.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:12:04.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:12:04.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:12:04.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:12:05.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:12:05.102 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:12:05.102 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:12:05.102 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:12:05.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:12:05.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:12:05.633 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:12:06.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:12:06.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:12:06.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-01T03:12:06.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:12:06.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-01T03:12:09.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:12:09.413 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:12:09.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:12:09.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:12:09.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:12:09.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:12:09.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:12:09.414 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:12:09.801 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:12:09.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:12:09.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:12:09.802 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:12:10.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:12:10.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:12:10.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:12:10.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:12:10.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:12:10.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:12:10.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:12:10.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:12:10.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:12:10.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:12:10.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:12:10.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-01T03:12:10.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:12:10.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-01T03:12:17.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:12:17.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:12:17.304 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:12:17.304 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:12:17.304 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:12:17.304 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:12:17.305 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:12:17.305 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:12:17.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:12:17.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:12:17.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:12:17.694 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:12:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:12:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:12:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:12:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:12:18.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:12:18.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:12:18.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:12:18.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:12:18.223 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:12:18.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:12:18.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:12:18.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-01T03:12:18.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-01T03:12:18.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-01T03:12:26.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-01T03:12:26.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:12:26.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:12:26.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:12:26.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:12:26.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:12:26.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:12:26.066 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:12:26.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-01T03:12:26.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:12:26.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:12:26.455 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-01T03:12:26.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-01T03:12:27.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-01T03:12:27.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-01T03:12:27.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-01T03:12:27.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-01T03:12:28.093 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_fingerprint": "7715cda0-390e-4300-9c1d-4cd9d4cc77c1", 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "version": "19.3.0-4640-g50288870", 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "commit": "5028887063e8eb518c95117479aa913eaa12ac8e", 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "timestamp": "2024-09-01T03:12:28.091785+0000", 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "tag": "", 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "health": { 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "status": "HEALTH_WARN", 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "checks": { 2024-09-01T03:12:28.094 INFO:tasks.workunit.client.0.smithi079.stdout: "OSDMAP_FLAGS": { 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 17 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [], 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "OSD_DOWN": { 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:12:28.095 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 osds down", 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "osd.1 (root=default,host=smithi079) is down" 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "PG_DEGRADED": { 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:12:28.096 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "severity": "HEALTH_WARN", 2024-09-01T03:12:28.097 INFO:tasks.workunit.client.0.smithi079.stdout: "summary": { 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: "count": 1 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: "detail": [ 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "application not enabled on pool 'test'" 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.098 INFO:tasks.workunit.client.0.smithi079.stdout: "muted": false 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "mutes": [] 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_first_committed": 1, 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap_last_committed": 1, 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "monmap": { 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:08:02.038114Z", 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:08:02.038114Z", 2024-09-01T03:12:28.099 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release": 19, 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "min_mon_release_name": "squid", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "election_strategy": 1, 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "disallowed_leaders: ": "", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": false, 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "tiebreaker_mon": "", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_ranks: ": "", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "features": { 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "persistent": [ 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "kraken", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "luminous", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "mimic", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap-prune", 2024-09-01T03:12:28.100 INFO:tasks.workunit.client.0.smithi079.stdout: "nautilus", 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "octopus", 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "pacific", 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "elector-pinging", 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "quincy", 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "reef", 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "squid" 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "optional": [] 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "mons": [ 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "rank": 0, 2024-09-01T03:12:28.101 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "a", 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115", 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 0 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:7115/0", 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "priority": 0, 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 0, 2024-09-01T03:12:28.102 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_location": "{}" 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: "quorum": [ 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: 0 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap": { 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 51, 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: "fsid": "d2f1e513-ecaf-447f-9f16-9352bc5d6107", 2024-09-01T03:12:28.103 INFO:tasks.workunit.client.0.smithi079.stdout: "created": "2024-09-01T03:08:02.942841+0000", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:11:51.771433+0000", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "last_up_change": "2024-09-01T03:11:51.377182+0000", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "last_in_change": "2024-09-01T03:11:00.067410+0000", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_num": 5800384, 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_set": [ 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "nodown", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "noin", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "noout", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "pglog_hardlimit", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snapdirs", 2024-09-01T03:12:28.104 INFO:tasks.workunit.client.0.smithi079.stdout: "recovery_deletes", 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "sortbitwise" 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_version": 12, 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "full_ratio": 0.99000000953674316, 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_snapshot": "", 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_max": 1, 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "max_osd": 3, 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "require_min_compat_client": "luminous", 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "min_compat_client": "jewel", 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "require_osd_release": "squid", 2024-09-01T03:12:28.105 INFO:tasks.workunit.client.0.smithi079.stdout: "allow_crimson": false, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "pools": [ 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_name": "test", 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "create_time": "2024-09-01T03:09:51.079565+0000", 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "flags": 16389, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "size": 3, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "min_size": 2, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_rule": 1, 2024-09-01T03:12:28.106 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_count": 0, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_target": 0, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_barrier": 0, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "is_stretch_pool": false, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "object_hash": 2, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_autoscale_mode": "off", 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num": 1, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num": 1, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_placement_num_target": 1, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_target": 1, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_num_pending": 1, 2024-09-01T03:12:28.107 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pg_merge_meta": { 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "source_pgid": "0.0", 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "ready_epoch": 0, 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_started": 0, 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": 0, 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "source_version": "0'0", 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "target_version": "0'0" 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "last_change": "30", 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend": "0", 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-01T03:12:28.108 INFO:tasks.workunit.client.0.smithi079.stdout: "last_force_op_resend_preluminous": "0", 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "auid": 0, 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_mode": "pool", 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_seq": 1, 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_epoch": 30, 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_snaps": [ 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "snapid": 1, 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "stamp": "2024-09-01T03:10:35.329885+0000", 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "snap1" 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps": "[]", 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_bytes": 0, 2024-09-01T03:12:28.109 INFO:tasks.workunit.client.0.smithi079.stdout: "quota_max_objects": 0, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "tiers": [], 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "tier_of": -1, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "read_tier": -1, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "write_tier": -1, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_mode": "none", 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_bytes": 0, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "target_max_objects": 0, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_flush_age": 0, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "cache_min_evict_age": 0, 2024-09-01T03:12:28.110 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profile": "myprofile", 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_params": { 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "none" 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_period": 0, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_count": 0, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "use_gmt_hitset": true, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "min_read_recency_for_promote": 0, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "min_write_recency_for_promote": 0, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_grade_decay_rate": 0, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "hit_set_search_last_n": 0, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "grade_table": [], 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "stripe_width": 8192, 2024-09-01T03:12:28.111 INFO:tasks.workunit.client.0.smithi079.stdout: "expected_num_objects": 0, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "fast_read": false, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "options": {}, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "application_metadata": {} 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "osds": [ 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "9f792487-0fd9-486c-914e-5de13226be1c", 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:12:28.112 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 37, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 43, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 46, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 50, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 45, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6802", 2024-09-01T03:12:28.113 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6803", 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.114 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6804", 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6805", 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.115 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6808", 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6809", 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.116 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.117 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:12:28.117 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.117 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6806", 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.120 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6807", 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3002794064 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6803/3002794064", 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6805/3002794064", 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3002794064", 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3002794064", 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.121 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "8569ecdc-ab13-4c87-a722-5e1735ac0930", 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 0, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 0, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 0, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 10, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 25, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 32, 2024-09-01T03:12:28.122 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.123 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.124 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.125 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.126 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 4070221926 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/4070221926", 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/4070221926", 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4070221926", 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4070221926", 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: "exists" 2024-09-01T03:12:28.127 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "uuid": "5268afd7-f8a5-4e5f-970d-389d79a07cb5", 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 1, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "in": 1, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 1, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_affinity": 1, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_begin": 39, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "last_clean_end": 43, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 50, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "up_thru": 0, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "down_at": 49, 2024-09-01T03:12:28.128 INFO:tasks.workunit.client.0.smithi079.stdout: "lost_at": 0, 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addrs": { 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6810", 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6811", 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.129 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addrs": { 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6812", 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6813", 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.130 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addrs": { 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6816", 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6817", 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.131 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addrs": { 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "addrvec": [ 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v2", 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6814", 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "v1", 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "addr": "127.0.0.1:6815", 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "nonce": 3773227636 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.132 INFO:tasks.workunit.client.0.smithi079.stdout: "public_addr": "127.0.0.1:6811/3773227636", 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "cluster_addr": "127.0.0.1:6813/3773227636", 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3773227636", 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3773227636", 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "state": [ 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "exists", 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "up" 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_xinfo": [ 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.133 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:11:49.301959+0000", 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:08:36.398738+0000", 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:10:46.890767+0000", 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:12:28.134 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:09:11.716841+0000", 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "down_stamp": "2024-09-01T03:11:51.228600+0000", 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_probability": 0, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "laggy_interval": 0, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "features": 4540719139924082687, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "old_weight": 0, 2024-09-01T03:12:28.135 INFO:tasks.workunit.client.0.smithi079.stdout: "last_purged_snaps_scrub": "2024-09-01T03:09:46.990184+0000", 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "dead_epoch": 0 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap": [], 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_items": [], 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_upmap_primaries": [], 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "pg_temp": [], 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "primary_temp": [], 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "blocklist": {}, 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "range_blocklist": {}, 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "erasure_code_profiles": { 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "default": { 2024-09-01T03:12:28.136 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "2", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van" 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "myprofile": { 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-device-class": "", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-failure-domain": "osd", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-num-failure-domains": "0", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-osds-per-failure-domain": "0", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "crush-root": "default", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "k": "2", 2024-09-01T03:12:28.137 INFO:tasks.workunit.client.0.smithi079.stdout: "m": "1", 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "plugin": "jerasure", 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "technique": "reed_sol_van", 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "w": "8" 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "removed_snaps_queue": [], 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "new_removed_snaps": [], 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "new_purged_snaps": [], 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "crush_node_flags": {}, 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "device_class_flags": {}, 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode": { 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_enabled": false, 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_bucket_count": 0, 2024-09-01T03:12:28.138 INFO:tasks.workunit.client.0.smithi079.stdout: "degraded_stretch_mode": 0, 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "recovering_stretch_mode": 0, 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "stretch_mode_bucket": 0 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_metadata": [ 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6804/3002794064,v1:127.0.0.1:6805/3002794064]", 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:12:28.139 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:12:28.140 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:08:08.986284Z", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:12:28.141 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6802/3002794064,v1:127.0.0.1:6803/3002794064]", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3002794064,v1:127.0.0.1:6809/3002794064]", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3002794064,v1:127.0.0.1:6807/3002794064]", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/0", 2024-09-01T03:12:28.142 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/4070221926,v1:127.0.0.1:6813/4070221926]", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:12:28.143 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:12:28.144 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:08:43.518248Z", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:12:28.145 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/4070221926,v1:127.0.0.1:6811/4070221926]", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4070221926,v1:127.0.0.1:6817/4070221926]", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4070221926,v1:127.0.0.1:6815/4070221926]", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:12:28.146 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/1", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "arch": "x86_64", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "back_addr": "[v2:127.0.0.1:6812/3773227636,v1:127.0.0.1:6813/3773227636]", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "back_iface": "", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs": "1", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_db": "0", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_dedicated_wal": "0", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "bluefs_single_shared_device": "1", 2024-09-01T03:12:28.147 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_allocation_from_file": "0", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_access_mode": "file", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_block_size": "4096", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_devices": "sda", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_rotational": "1", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_size": "107374182400", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_support_discard": "1", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_bdev_type": "hdd", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "bluestore_min_alloc_size": "4096", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_release": "squid", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:12:28.148 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_short": "19.3.0-4640-g50288870", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "ceph_version_when_created": "ceph version 19.3.0-4640-g50288870 (5028887063e8eb518c95117479aa913eaa12ac8e) squid (dev)", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "created_at": "2024-09-01T03:09:18.120448Z", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "default_device_class": "hdd", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5QDP6", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": "sda", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "distro": "centos", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_description": "CentOS Stream 9", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "distro_version": "9", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "front_addr": "[v2:127.0.0.1:6810/3773227636,v1:127.0.0.1:6811/3773227636]", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "front_iface": "", 2024-09-01T03:12:28.149 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3773227636,v1:127.0.0.1:6817/3773227636]", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3773227636,v1:127.0.0.1:6815/3773227636]", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "hostname": "smithi079", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "journal_rotational": "1", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Aug 22 17:03:23 UTC 2024", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "kernel_version": "5.14.0-503.el9.x86_64", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_swap_kb": "0", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "mem_total_kb": "32487348", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "os": "Linux", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_data": "td/divergent-priors/2", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_objectstore": "bluestore", 2024-09-01T03:12:28.150 INFO:tasks.workunit.client.0.smithi079.stdout: "osdspec_affinity": "", 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "rotational": "1" 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_clean_epochs": { 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "min_last_epoch_clean": 26, 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "last_epoch_clean": { 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "per_pool": [ 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: "floor": 26 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.151 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_epochs": [ 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 46 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 50 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.152 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_first_committed": 1, 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "osdmap_last_committed": 51, 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "crushmap": { 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "devices": [ 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.0", 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.1", 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:12:28.153 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd.2", 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "class": "hdd" 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "types": [ 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 0, 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "osd" 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.154 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "host" 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 2, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "chassis" 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 3, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "rack" 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 4, 2024-09-01T03:12:28.155 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "row" 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 5, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pdu" 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 6, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "pod" 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 7, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "room" 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.156 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 8, 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "datacenter" 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 9, 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "zone" 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 10, 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "region" 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.157 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "root" 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "buckets": [ 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -1, 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default", 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:12:28.158 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -2, 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "default~hdd", 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 11, 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "root", 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:12:28.159 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -3, 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079", 2024-09-01T03:12:28.160 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.161 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "id": -4, 2024-09-01T03:12:28.162 INFO:tasks.workunit.client.0.smithi079.stdout: "name": "smithi079~hdd", 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "type_id": 1, 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "type_name": "host", 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 19206, 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "alg": "straw2", 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "hash": "rjenkins1", 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "items": [ 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 0, 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 0 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.163 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 1, 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 1 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: "id": 2, 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: "weight": 6402, 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: "pos": 2 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: "rules": [ 2024-09-01T03:12:28.164 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 0, 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "replicated_rule", 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 1, 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_firstn", 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:12:28.165 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_id": 1, 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: "rule_name": "test", 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: "type": 3, 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: "steps": [ 2024-09-01T03:12:28.166 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_chooseleaf_tries", 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 5 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "set_choose_tries", 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 100 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "take", 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "item": -1, 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: "item_name": "default" 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.167 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.168 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "choose_indep", 2024-09-01T03:12:28.168 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 0, 2024-09-01T03:12:28.168 INFO:tasks.workunit.client.0.smithi079.stdout: "type": "osd" 2024-09-01T03:12:28.168 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.168 INFO:tasks.workunit.client.0.smithi079.stderr:report 3672240686 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "op": "emit" 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: ] 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "tunables": { 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_tries": 0, 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_local_fallback_tries": 0, 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_total_tries": 50, 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_descend_once": 1, 2024-09-01T03:12:28.169 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_vary_r": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "chooseleaf_stable": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_descents": 100, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "msr_collision_tries": 100, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "straw_calc_version": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "allowed_bucket_algs": 54, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "profile": "jewel", 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "optimal_tunables": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "legacy_tunables": 0, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "minimum_required_version": "jewel", 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables2": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v2_rules": 1, 2024-09-01T03:12:28.170 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables3": 1, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v3_rules": 0, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v4_buckets": 1, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "require_feature_tunables5": 1, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "has_v5_rules": 0, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "has_msr_rules": 0 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "choose_args": {} 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "fsmap": { 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 1, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "btime": "2024-09-01T03:08:02:918133+0000", 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "default_fscid": -1, 2024-09-01T03:12:28.171 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": { 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "compat": {}, 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "ro_compat": {}, 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "incompat": { 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_1": "base v0.20", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_2": "client writeable ranges", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_3": "default file layouts on dirs", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_4": "dir inode in separate object", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_5": "mds uses versioned encoding", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_8": "no anchor table", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_9": "file layout v2", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_10": "snaprealm v2", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_11": "minor log segments", 2024-09-01T03:12:28.172 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_12": "quiesce subvolumes" 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "feature_flags": { 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "enable_multiple": true, 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "ever_enabled_multiple": true 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "standbys": [], 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "filesystems": [] 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_first_committed": 1, 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "mdsmap_last_committed": 1, 2024-09-01T03:12:28.173 INFO:tasks.workunit.client.0.smithi079.stdout: "auth": { 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 5, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_secrets": 3 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_active": 1, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_unknown": 0, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osd": 3, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_sum": { 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48454, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:12:28.174 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-09-01T03:12:28.175 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:12:28.176 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:12:28.177 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:12:28.178 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 0 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum": { 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:12:28.179 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 85112, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2292, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 15, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 82736, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314487688, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322035392512, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:12:28.180 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2347008, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1605609, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 16257, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 84721791 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:12:28.181 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_sum_by_class": { 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "hdd": { 2024-09-01T03:12:28.182 INFO:tasks.workunit.client.0.smithi079.stdout: "up_from": 0, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 0, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pgs": 3, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "num_osds": 3, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_osds": 3, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "num_per_pool_omap_osds": 3, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "kb": 314572800, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used": 85112, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_data": 2292, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_omap": 15, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_used_meta": 82736, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "kb_avail": 314487688, 2024-09-01T03:12:28.183 INFO:tasks.workunit.client.0.smithi079.stdout: "statfs": { 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 322122547200, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 322035392512, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 2347008, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1605609, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 16257, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 84721791 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "hb_peers": [], 2024-09-01T03:12:28.184 INFO:tasks.workunit.client.0.smithi079.stdout: "snap_trim_queue_len": 0, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "num_snap_trimming": 0, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shards_repaired": 0, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "op_queue_age_hist": { 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "histogram": [], 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "upper_bound": 1 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "perf_stat": { 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ms": 0, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ms": 0, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "commit_latency_ns": 0, 2024-09-01T03:12:28.185 INFO:tasks.workunit.client.0.smithi079.stdout: "apply_latency_ns": 0 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "alerts": [], 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "network_ping_times": [] 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "pool_stats": [ 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "poolid": 1, 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg": 1, 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "stat_sum": { 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes": 48454, 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects": 101, 2024-09-01T03:12:28.186 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_clones": 1, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_object_copies": 303, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing_on_primary": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_missing": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_degraded": 101, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_misplaced": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_unfound": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_dirty": 101, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_whiteouts": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_read_kb": 0, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write": 101, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_write_kb": 101, 2024-09-01T03:12:28.187 INFO:tasks.workunit.client.0.smithi079.stdout: "num_scrub_errors": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_shallow_scrub_errors": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_deep_scrub_errors": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_recovered": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_recovered": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_keys_recovered": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_omap": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_hit_set_archive": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_bytes_hit_set_archive": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_kb": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_kb": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_promote": 0, 2024-09-01T03:12:28.188 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_high": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_flush_mode_low": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_some": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_evict_mode_full": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_pinned": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_legacy_snapsets": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_large_omap_objects": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_manifest": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_bytes": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_omap_keys": 0, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "num_objects_repaired": 0 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "store_stats": { 2024-09-01T03:12:28.189 INFO:tasks.workunit.client.0.smithi079.stdout: "total": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "available": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "internally_reserved": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "allocated": 1241088, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "data_stored": 1241088, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_allocated": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "data_compressed_original": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "omap_allocated": 0, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "internal_metadata": 0 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "log_size": 18, 2024-09-01T03:12:28.190 INFO:tasks.workunit.client.0.smithi079.stdout: "ondisk_log_size": 18, 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "up": 3, 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "acting": 3, 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "num_store_stats": 3 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "osd_stats": [ 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 197568495626 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 42949672990 2024-09-01T03:12:28.191 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: "seq": 214748364810 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_state": [ 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: "state": "active+undersized+degraded", 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: "num": 1 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.192 INFO:tasks.workunit.client.0.smithi079.stdout: "num_pg_by_osd": [ 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2147483647, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 2, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.193 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 0, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 1, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 1, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "osd": 1, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "num_primary_pg": 0, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "num_acting_pg": 0, 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "num_up_not_acting_pg": 0 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": [ 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: { 2024-09-01T03:12:28.194 INFO:tasks.workunit.client.0.smithi079.stdout: "pool": 1, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "purged_snaps": {} 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: ], 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "servicemap": { 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "epoch": 5, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "modified": "2024-09-01T03:10:49.666290+0000", 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "services": {} 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: }, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_first_committed": 1, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "mgrstat_last_committed": 122, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_first_committed": 1, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "logm_last_committed": 488, 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "paxos": { 2024-09-01T03:12:28.195 INFO:tasks.workunit.client.0.smithi079.stdout: "first_committed": 1, 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stdout: "last_committed": 666, 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stdout: "last_pn": 0, 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stdout: "accepted_pn": 0 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stdout: } 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-01T03:12:28.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-01T03:12:28.197 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery 2024-09-01T03:12:28.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:28.472 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:28.473 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 51:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:28.473 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:28.473 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:28.473 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:28.483 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.0.asok 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-01T03:12:28.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.0.asok set_recovery_delay 100000 2024-09-01T03:12:28.566 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:12:28.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-01T03:12:28.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-01T03:12:28.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-01T03:12:28.573 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-01T03:12:28.574 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:12:28.574 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:12:28.574 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:12:28.575 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.2.asok 2024-09-01T03:12:28.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-01T03:12:28.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.2.asok set_recovery_delay 100000 2024-09-01T03:12:28.656 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:12:28.663 INFO:tasks.workunit.client.0.smithi079.stdout:revive divergent 1 2024-09-01T03:12:28.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-01T03:12:28.664 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:29.032 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:29.032 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 51:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:29.032 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:29.032 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:29.033 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:29.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-01T03:12:29.626 INFO:tasks.workunit.client.0.smithi079.stderr:noup is set 2024-09-01T03:12:29.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-01T03:12:29.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:12:29.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:12:29.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:12:29.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:12:29.642 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:12:29.642 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:12:29.642 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:12:29.642 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:12:29.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:12:29.644 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:12:29.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:12:29.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:12:29.645 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T03:12:29.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:12:29.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:12:29.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:12:29.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:12:29.676 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:29.674+0000 7f33d9756640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:12:29.685 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:29.684+0000 7f33d9756640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:12:29.687 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:29.685+0000 7f33d9756640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:12:30.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-01T03:12:30.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-01T03:12:30.524 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:30.523+0000 7f33d9756640 -1 Falling back to public interface 2024-09-01T03:12:33.121 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:33.120+0000 7f33d9756640 -1 osd.1 30 log_to_monitors true 2024-09-01T03:12:35.109 INFO:tasks.workunit.client.0.smithi079.stdout:delay recovery divergent 2024-09-01T03:12:35.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-01T03:12:35.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:35.488 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:35.489 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 51:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:35.489 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:35.489 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:35.489 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:35.500 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-01T03:12:35.500 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.22149/ceph-osd.1.asok 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-01T03:12:35.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.22149/ceph-osd.1.asok set_recovery_delay 100000 2024-09-01T03:12:35.581 INFO:tasks.workunit.client.0.smithi079.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-01T03:12:35.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-01T03:12:36.212 INFO:tasks.workunit.client.0.smithi079.stderr:noup is unset 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:12:36.224 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:12:36.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:12:36.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:12:36.669 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/3002794064,v1:127.0.0.1:6803/3002794064] [v2:127.0.0.1:6804/3002794064,v1:127.0.0.1:6805/3002794064] exists,up 9f792487-0fd9-486c-914e-5de13226be1c 2024-09-01T03:12:36.670 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:12:36.671 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:12:36.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:12:36.672 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:12:37.060 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/1130954838,v1:127.0.0.1:6819/1130954838] [v2:127.0.0.1:6820/1130954838,v1:127.0.0.1:6821/1130954838] exists,up 8569ecdc-ab13-4c87-a722-5e1735ac0930 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:12:37.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:12:37.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:12:37.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:12:37.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:12:37.063 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:12:37.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:12:37.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:12:37.450 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/3773227636,v1:127.0.0.1:6811/3773227636] [v2:127.0.0.1:6812/3773227636,v1:127.0.0.1:6813/3773227636] exists,up 5268afd7-f8a5-4e5f-970d-389d79a07cb5 2024-09-01T03:12:37.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:12:37.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:12:37.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:12:37.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:37.819 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:37.819 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 53:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:37.820 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:37.820 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:37.820 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:37.830 INFO:tasks.workunit.client.0.smithi079.stdout:wait for peering 2024-09-01T03:12:37.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-01T03:12:37.830 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:38.220 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:38.221 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 53:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:38.221 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:38.221 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:38.221 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:38.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-01T03:12:38.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-01T03:12:38.346 INFO:tasks.workunit.client.0.smithi079.stdout:killing divergent 1 2024-09-01T03:12:38.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:38.710 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:38.711 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 53:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:38.711 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:38.711 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:38.711 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:38.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-01T03:12:38.722 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:12:38.722 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:12:38.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:12:38.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:12:38.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:12:38.827 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:12:38.828 INFO:tasks.workunit.client.0.smithi079.stdout:reviving divergent 1 2024-09-01T03:12:38.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-01T03:12:38.828 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:12:39.206 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:12:39.207 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-01T03:11:52.132150+0000 43'102 53:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T12:21:11.476905+0000 0 0 2024-09-01T03:12:39.207 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:12:39.207 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:12:39.207 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:12:39.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:12:39.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:12:39.222 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:12:39.223 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-01T03:12:39.224 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:12:39.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:12:39.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=d2f1e513-ecaf-447f-9f16-9352bc5d6107 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.22149/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:12:39.225 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-01T03:12:39.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:12:39.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:12:39.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:12:39.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:12:39.256 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:39.255+0000 7f2672c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:12:39.258 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:39.257+0000 7f2672c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:12:39.259 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:39.258+0000 7f2672c11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:12:39.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:12:39.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:12:39.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:12:39.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:12:39.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:12:39.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:12:39.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:12:39.651 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:12:39.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:12:39.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:12:40.102 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/1130954838,v1:127.0.0.1:6819/1130954838] [v2:127.0.0.1:6820/1130954838,v1:127.0.0.1:6821/1130954838] exists,up 8569ecdc-ab13-4c87-a722-5e1735ac0930 2024-09-01T03:12:40.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:12:40.102 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:12:40.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:12:40.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-01T03:12:41.097 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:41.095+0000 7f2672c11640 -1 Falling back to public interface 2024-09-01T03:12:43.824 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:12:43.823+0000 7f2672c11640 -1 osd.1 57 log_to_monitors true 2024-09-01T03:13:00.103 INFO:tasks.workunit.client.0.smithi079.stdout:allowing recovery 2024-09-01T03:13:00.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-01T03:13:00.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:13:00.479 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:13:00.479 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-01T03:12:45.560828+0000 57'103 62:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T08:24:20.112983+0000 0 0 2024-09-01T03:13:00.479 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:13:00.479 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:13:00.479 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:13:00.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-01T03:13:00.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-01T03:13:00.601 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:13:00.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-01T03:13:00.610 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-01T03:13:00.723 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:13:00.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-01T03:13:00.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-01T03:13:00.844 INFO:tasks.workunit.client.0.smithi079.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-01T03:13:00.852 INFO:tasks.workunit.client.0.smithi079.stdout:reading divergent objects 2024-09-01T03:13:00.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-01T03:13:00.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-01T03:13:01.235 INFO:tasks.workunit.client.0.smithi079.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-01T03:13:01.236 INFO:tasks.workunit.client.0.smithi079.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-01T03:12:45.560828+0000 57'103 62:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-01T03:09:51.150465+0000 0'0 2024-09-01T03:09:51.150465+0000 0 0 periodic scrub scheduled @ 2024-09-02T08:24:20.112983+0000 0 0 2024-09-01T03:13:01.236 INFO:tasks.workunit.client.0.smithi079.stdout: 2024-09-01T03:13:01.236 INFO:tasks.workunit.client.0.smithi079.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-01T03:13:01.236 INFO:tasks.workunit.client.0.smithi079.stderr:dumped pgs 2024-09-01T03:13:01.247 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-01T03:13:01.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-01T03:13:01.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:13:01.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-01T03:13:01.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:13:01.289 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-01T03:13:01.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:13:01.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-01T03:13:01.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-01T03:13:01.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-01T03:13:01.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-01T03:13:01.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-01T03:13:01.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:13:01.404 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-01T03:10:27.785677+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:13:01.404 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-01T03:10:27.785677+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-01T03:13:01.404 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-01T03:10:35.552789+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-01T03:10:35.552789+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-09-01T03:13:01.405 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-01T03:10:35.552789+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.406 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-01T03:13:01.406 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-01T03:10:34.324408+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-01T03:10:35.330671+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.406 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-01T03:10:34.324408+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.406 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-01T03:10:35.330671+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.406 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-01T03:13:01.406 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-09-01T03:13:01.407 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-01T03:13:01.407 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-01T03:13:01.407 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-09-01T03:13:01.407 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-01T03:10:35.330671+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-01T03:13:01.407 INFO:tasks.workunit.client.0.smithi079.stdout:td/divergent-priors/osd.1.log:2024-09-01T03:12:37.325+0000 7f33bd6c8640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-01T03:13:01.407 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-01T03:13:01.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-01T03:13:01.414 INFO:tasks.workunit.client.0.smithi079.stdout:success 2024-09-01T03:13:01.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-01T03:13:01.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-01T03:13:01.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-01T03:13:01.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-01T03:13:01.985 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'test' does not exist 2024-09-01T03:13:01.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-01T03:13:01.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:13:01.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:13:01.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:13:01.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:13:01.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:13:02.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:13:02.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:13:02.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:13:02.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:13:02.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:13:02.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:13:02.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:13:02.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:13:02.314 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:13:02.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:13:02.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:13:02.315 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:13:02.315 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:13:02.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:13:02.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:13:02.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T03:13:02.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:13:02.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:02.340 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:13:02.340 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T03:13:02.341 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/divergent-priors 0 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:13:02.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:13:02.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:13:02.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:13:02.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:13:02.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:13:02.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:13:02.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:13:02.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:13:02.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:13:02.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:13:02.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:13:02.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:13:02.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:13:02.347 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:13:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:13:02.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T03:13:02.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-01T03:13:02.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:13:02.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:02.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.22149 2024-09-01T03:13:02.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.22149 2024-09-01T03:13:02.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:13:02.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:13:02.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T03:13:02.352 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T03:13:02.352 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T03:13:02.379 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-01T03:13:02.379 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-01T03:13:02.442 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T03:13:02.447 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T03:13:02.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T03:13:02.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T03:13:02.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ec-error-rollforward 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-01T03:13:02.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none ' 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-01T03:13:02.449 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ec-error-rollforward 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ec-error-rollforward 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:13:02.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:13:02.452 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:13:02.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:13:02.453 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:13:02.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:13:02.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:13:02.453 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:13:02.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:13:02.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:13:02.454 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:13:02.454 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:13:02.455 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:13:02.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:13:02.456 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-01T03:13:02.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:13:02.458 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:02.459 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:02.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.79672 2024-09-01T03:13:02.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:13:02.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:13:02.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ec-error-rollforward 2024-09-01T03:13:02.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:13:02.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:02.460 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:02.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.79672 2024-09-01T03:13:02.461 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ec-error-rollforward 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:13:02.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ec-error-rollforward/a 2024-09-01T03:13:02.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:13:02.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:13:03.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:13:03.094 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:03.095 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.79672/ceph-mon.a.asok 2024-09-01T03:13:03.096 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:13:03.096 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.79672/ceph-mon.a.asok config get fsid 2024-09-01T03:13:03.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:13:03.174 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:13:03.175 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:03.176 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:03.176 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.79672/ceph-mon.a.asok 2024-09-01T03:13:03.176 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:13:03.176 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.79672/ceph-mon.a.asok config get mon_host 2024-09-01T03:13:03.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-01T03:13:03.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ec-error-rollforward 2024-09-01T03:13:03.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:13:03.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:13:03.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:13:03.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ec-error-rollforward/x 2024-09-01T03:13:03.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:13:04.098 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:13:04.098 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:13:04.098 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:13:04.098 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:13:04.098 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:04.098 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:04.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:13:04.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:13:04.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:13:04.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-01T03:13:04.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:04.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-01T03:13:04.131 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:13:04.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4b839a65-e33c-4a01-bf61-599111c9c6ac 2024-09-01T03:13:04.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4b839a65-e33c-4a01-bf61-599111c9c6ac' 2024-09-01T03:13:04.132 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 4b839a65-e33c-4a01-bf61-599111c9c6ac 2024-09-01T03:13:04.132 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:13:04.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDA29NmG8aVCBAABhmrDyVQDBS4e0iL9v4kHQ== 2024-09-01T03:13:04.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDA29NmG8aVCBAABhmrDyVQDBS4e0iL9v4kHQ=="}' 2024-09-01T03:13:04.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4b839a65-e33c-4a01-bf61-599111c9c6ac -i td/ec-error-rollforward/0/new.json 2024-09-01T03:13:04.434 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:13:04.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-01T03:13:04.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDA29NmG8aVCBAABhmrDyVQDBS4e0iL9v4kHQ== --osd-uuid 4b839a65-e33c-4a01-bf61-599111c9c6ac 2024-09-01T03:13:04.471 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:04.470+0000 7f721ee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:04.503 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:04.501+0000 7f721ee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:04.503 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:04.502+0000 7f721ee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:04.504 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:04.502+0000 7f721ee16640 -1 bdev(0x55fd6ccd5c00 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:13:04.504 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:04.503+0000 7f721ee16640 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-01T03:13:09.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-01T03:13:09.992 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:13:09.993 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:13:09.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:13:09.993 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:13:10.520 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:13:10.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:13:10.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:13:10.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:13:10.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:13:10.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:13:10.545 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:10.543+0000 7f58a36a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:10.557 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:10.556+0000 7f58a36a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:10.558 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:10.557+0000 7f58a36a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:10.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:13:10.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:13:10.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:13:10.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:13:10.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:13:10.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:10.964 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:13:10.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:13:10.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:10.965 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:11.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:11.649 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:11.648+0000 7f58a36a5640 -1 Falling back to public interface 2024-09-01T03:13:12.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:12.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:12.398 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:13:12.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:13:12.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:12.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:12.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:13.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:13.825 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:13.825 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:13:13.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:13:13.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:13.826 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:14.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:14.385 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:14.383+0000 7f58a36a5640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:13:15.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:15.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:15.208 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:13:15.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:13:15.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:15.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:15.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:16.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:16.743 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:16.743 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:13:16.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:13:16.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:16.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:17.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:18.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:18.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:18.214 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:13:18.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:13:18.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:18.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:18.633 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:19.634 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:19.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:19.635 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:13:19.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:13:19.636 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:19.637 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:20.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:21.049 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:21.049 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:13:21.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:13:21.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:21.050 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:21.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:22.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:22.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:22.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:13:22.565 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:13:22.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:22.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:22.983 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:23.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:23.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:23.985 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:13:23.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:13:23.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:23.986 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:24.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:25.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:25.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:25.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:13:25.380 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:13:25.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:25.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:25.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:26.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:26.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:26.767 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:13:26.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:13:26.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:26.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:27.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:28.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:28.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:28.228 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:13:28.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:13:28.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:28.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:28.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:29.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:13:29.734 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:13:29.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:29.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:30.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:31.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:31.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:31.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:13:31.309 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:13:31.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:31.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:32.144 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:33.145 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:33.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:33.146 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:13:33.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:13:33.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:33.146 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:33.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:34.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:34.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:34.746 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:13:34.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:13:34.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:34.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:35.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:36.732 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:36.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:36.733 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:13:36.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:13:36.733 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:36.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:37.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:38.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:38.298 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:38.299 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T03:13:38.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T03:13:38.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:38.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:13:38.791 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1995305368,v1:127.0.0.1:6803/1995305368] [v2:127.0.0.1:6804/1995305368,v1:127.0.0.1:6805/1995305368] exists,up 4b839a65-e33c-4a01-bf61-599111c9c6ac 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:13:38.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:13:38.796 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:13:38.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:13:38.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:13:38.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:13:38.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-01T03:13:38.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:13:38.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f7fc88ec-09ea-4356-8af5-140662ab416b 2024-09-01T03:13:38.798 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 f7fc88ec-09ea-4356-8af5-140662ab416b 2024-09-01T03:13:38.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f7fc88ec-09ea-4356-8af5-140662ab416b' 2024-09-01T03:13:38.799 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:13:38.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDi29NmiBZRMBAAveVvBxsVB2kwlt0bezTQ6g== 2024-09-01T03:13:38.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDi29NmiBZRMBAAveVvBxsVB2kwlt0bezTQ6g=="}' 2024-09-01T03:13:38.812 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f7fc88ec-09ea-4356-8af5-140662ab416b -i td/ec-error-rollforward/1/new.json 2024-09-01T03:13:39.306 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:13:39.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-01T03:13:39.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDi29NmiBZRMBAAveVvBxsVB2kwlt0bezTQ6g== --osd-uuid f7fc88ec-09ea-4356-8af5-140662ab416b 2024-09-01T03:13:39.345 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:39.344+0000 7f1447a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:39.357 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:39.356+0000 7f1447a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:39.358 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:39.357+0000 7f1447a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:39.358 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:39.357+0000 7f1447a15640 -1 bdev(0x563fb9035c00 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-09-01T03:13:39.359 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:39.357+0000 7f1447a15640 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-01T03:13:45.418 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-01T03:13:45.419 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:13:45.419 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T03:13:45.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T03:13:45.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:13:45.948 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:13:45.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T03:13:45.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:13:45.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:13:45.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:13:45.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:13:45.972 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:45.971+0000 7faa58fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:45.992 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:45.991+0000 7faa58fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:45.993 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:45.992+0000 7faa58fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:13:46.404 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:13:46.405 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:13:46.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:46.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:46.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:47.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:47.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:47.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:13:47.781 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:13:47.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:47.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:47.844 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:47.843+0000 7faa58fbd640 -1 Falling back to public interface 2024-09-01T03:13:48.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:49.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:49.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:49.178 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:13:49.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:13:49.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:49.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:49.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:50.602 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:50.603 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:13:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:13:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:13:50.605+0000 7faa58fbd640 -1 osd.1 0 log_to_monitors true 2024-09-01T03:13:51.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:52.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:52.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:52.091 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:13:52.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:13:52.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:52.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:52.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:53.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:53.537 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:13:53.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:53.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:13:53.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:53.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:53.944 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:54.945 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:54.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:54.946 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:13:54.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:13:54.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:54.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:55.391 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:56.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:56.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:56.393 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:13:56.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:13:56.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:56.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:56.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:57.846 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:57.846 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:57.846 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:13:57.846 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:13:57.847 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:57.847 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:58.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:13:59.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:13:59.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:13:59.286 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:13:59.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:13:59.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:13:59.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:13:59.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:00.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:00.677 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:14:00.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:14:00.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:00.678 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:01.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:02.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:02.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:02.088 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:14:02.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:14:02.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:02.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:02.674 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:03.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:03.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:03.676 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:14:03.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:14:03.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:03.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:04.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:05.284 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:05.285 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:14:05.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:05.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:14:05.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:05.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:06.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:07.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:07.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:07.047 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:14:07.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:14:07.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:07.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:07.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:08.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:08.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:08.711 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:14:08.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:14:08.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:08.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:10.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:11.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:11.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:11.029 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:14:11.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:14:11.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:11.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:11.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:12.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:12.782 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:12.782 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:14:12.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:14:12.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:12.783 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:14:13.183 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/558364149,v1:127.0.0.1:6811/558364149] [v2:127.0.0.1:6812/558364149,v1:127.0.0.1:6813/558364149] exists,up f7fc88ec-09ea-4356-8af5-140662ab416b 2024-09-01T03:14:13.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:14:13.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:14:13.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:14:13.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-01T03:14:13.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-01T03:14:13.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:14:13.188 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:14:13.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=024208b2-e5b9-44b7-87c7-b91ae59b9da0 2024-09-01T03:14:13.190 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 024208b2-e5b9-44b7-87c7-b91ae59b9da0 2024-09-01T03:14:13.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 024208b2-e5b9-44b7-87c7-b91ae59b9da0' 2024-09-01T03:14:13.191 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:14:13.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAF3NNmt9YMDBAAjUQvt1gG/CKm8Jq85VFggA== 2024-09-01T03:14:13.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAF3NNmt9YMDBAAjUQvt1gG/CKm8Jq85VFggA=="}' 2024-09-01T03:14:13.204 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 024208b2-e5b9-44b7-87c7-b91ae59b9da0 -i td/ec-error-rollforward/2/new.json 2024-09-01T03:14:13.639 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:14:13.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-01T03:14:13.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAF3NNmt9YMDBAAjUQvt1gG/CKm8Jq85VFggA== --osd-uuid 024208b2-e5b9-44b7-87c7-b91ae59b9da0 2024-09-01T03:14:13.678 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:13.675+0000 7f6586c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:13.701 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:13.698+0000 7f6586c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:13.702 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:13.699+0000 7f6586c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:13.702 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:13.700+0000 7f6586c0d640 -1 bdev(0x55bc48d79c00 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-09-01T03:14:13.702 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:13.700+0000 7f6586c0d640 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-01T03:14:20.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-01T03:14:20.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:14:20.148 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T03:14:20.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T03:14:20.148 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:14:20.719 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:14:20.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T03:14:20.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:14:20.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:14:20.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:14:20.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:14:20.745 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:20.743+0000 7f1a76815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:20.764 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:20.762+0000 7f1a76815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:20.765 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:20.764+0000 7f1a76815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:21.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T03:14:21.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:14:21.169 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:14:21.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:14:21.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:14:21.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:21.170 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:14:21.170 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:14:21.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:21.171 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:21.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:22.105 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:22.104+0000 7f1a76815640 -1 Falling back to public interface 2024-09-01T03:14:22.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:22.588 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:22.589 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:14:22.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:14:22.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:22.589 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:23.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:24.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:24.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:24.007 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:14:24.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:14:24.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:24.007 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:24.448 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:25.076 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:25.075+0000 7f1a76815640 -1 osd.2 0 log_to_monitors true 2024-09-01T03:14:25.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:25.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:25.450 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:14:25.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:14:25.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:25.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:25.895 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:26.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:26.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:26.897 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:14:26.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:14:26.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:26.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:27.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:28.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:28.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:28.287 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:14:28.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:14:28.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:28.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:28.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:29.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:29.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:29.779 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:14:29.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:14:29.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:29.780 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:30.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:31.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:31.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:31.330 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:14:31.330 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:14:31.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:31.331 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:31.734 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:32.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:32.735 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:32.736 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:14:32.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:14:32.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:32.736 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:33.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:34.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:34.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:34.180 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:14:34.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:14:34.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:34.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:34.693 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:35.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:35.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:35.695 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:14:35.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:14:35.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:35.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:36.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:37.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:37.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:37.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:14:37.209 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:14:37.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:37.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:37.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:38.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:38.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:38.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:14:38.599 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:14:38.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:38.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:39.118 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:40.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:40.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:40.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:14:40.120 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:14:40.120 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:40.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:40.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:41.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:41.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:41.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:14:41.644 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:14:41.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:41.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:42.305 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:43.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:43.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:43.307 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:14:43.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:14:43.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:43.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:44.082 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:45.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:45.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:45.085 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:14:45.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:14:45.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:45.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:45.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:46.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:46.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:46.862 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:14:46.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:14:46.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:46.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:48.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:49.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:49.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:49.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T03:14:49.054 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T03:14:49.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:49.055 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:14:49.487 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3300469933,v1:127.0.0.1:6819/3300469933] [v2:127.0.0.1:6820/3300469933,v1:127.0.0.1:6821/3300469933] exists,up 024208b2-e5b9-44b7-87c7-b91ae59b9da0 2024-09-01T03:14:49.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:14:49.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:14:49.493 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:14:49.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8783d08d-aa7c-428c-a39b-7e5e0ea7e890 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stdout:add osd3 8783d08d-aa7c-428c-a39b-7e5e0ea7e890 2024-09-01T03:14:49.495 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 8783d08d-aa7c-428c-a39b-7e5e0ea7e890' 2024-09-01T03:14:49.496 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:14:49.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAp3NNmi7YrHhAAKUdgmMDysVmsCEjUCWpXJg== 2024-09-01T03:14:49.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAp3NNmi7YrHhAAKUdgmMDysVmsCEjUCWpXJg=="}' 2024-09-01T03:14:49.508 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8783d08d-aa7c-428c-a39b-7e5e0ea7e890 -i td/ec-error-rollforward/3/new.json 2024-09-01T03:14:50.052 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:14:50.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-01T03:14:50.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAp3NNmi7YrHhAAKUdgmMDysVmsCEjUCWpXJg== --osd-uuid 8783d08d-aa7c-428c-a39b-7e5e0ea7e890 2024-09-01T03:14:50.091 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:50.089+0000 7f44afe0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:50.104 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:50.101+0000 7f44afe0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:50.105 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:50.102+0000 7f44afe0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:50.106 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:50.102+0000 7f44afe0b640 -1 bdev(0x55a4d603dc00 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-09-01T03:14:50.106 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:50.102+0000 7f44afe0b640 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-01T03:14:56.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-01T03:14:56.066 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:14:56.067 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd3 key to auth repository 2024-09-01T03:14:56.067 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-01T03:14:56.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:14:56.602 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-09-01T03:14:56.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-01T03:14:56.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:14:56.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:14:56.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:14:56.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:14:56.629 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:56.626+0000 7f516f7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:56.659 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:56.656+0000 7f516f7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:56.660 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:56.657+0000 7f516f7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:14:57.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-01T03:14:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:14:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T03:14:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:14:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:14:57.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:57.023 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:14:57.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:14:57.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:57.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:14:57.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:57.499 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:14:57.497+0000 7f516f7bb640 -1 Falling back to public interface 2024-09-01T03:14:58.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:58.433 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:58.433 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:14:58.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:14:58.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:58.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:14:58.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:14:59.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:14:59.851 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:14:59.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:14:59.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:14:59.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:14:59.852 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:00.169 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:00.167+0000 7f516f7bb640 -1 osd.3 0 log_to_monitors true 2024-09-01T03:15:00.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:01.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:01.257 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:01.258 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:15:01.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:15:01.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:01.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:01.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:02.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:02.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:02.685 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:15:02.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:15:02.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:02.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:03.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:04.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:04.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:04.233 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:15:04.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:15:04.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:04.234 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:04.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:05.764 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:05.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:05.765 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:15:05.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:15:05.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:05.766 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:06.218 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:07.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:07.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:07.220 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:15:07.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:15:07.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:07.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:07.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:08.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:08.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:08.618 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:15:08.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:15:08.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:08.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:09.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:10.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:10.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:10.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:15:10.110 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:15:10.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:10.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:10.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:11.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:11.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:11.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:15:11.593 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:15:11.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:11.594 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:12.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:13.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:13.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:13.069 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:15:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:15:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:13.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:13.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:14.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:14.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:14.642 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:15:14.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:15:14.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:14.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:15.675 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:16.676 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:15:16.677 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:15:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:16.677 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:17.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:18.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:18.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:18.725 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:15:18.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:15:18.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:18.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:19.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:20.510 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:20.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:20.511 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:15:20.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:15:20.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:20.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:21.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:22.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:22.612 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:22.612 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:15:22.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:15:22.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:22.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:23.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:24.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:24.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:24.525 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:15:24.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:15:24.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:24.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:24.954 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3680603446,v1:127.0.0.1:6827/3680603446] [v2:127.0.0.1:6828/3680603446,v1:127.0.0.1:6829/3680603446] exists,up 8783d08d-aa7c-428c-a39b-7e5e0ea7e890 2024-09-01T03:15:24.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:15:24.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:15:24.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:15:24.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-01T03:15:25.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-01T03:15:26.473 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'ec' already exists 2024-09-01T03:15:26.485 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-01T03:15:28.201 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-01T03:15:28.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 84283 2024-09-01T03:15:28.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=87038 2024-09-01T03:15:28.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-01T03:15:28.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-01T03:15:29.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 87059' 2024-09-01T03:15:29.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-01T03:15:29.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 87060' 2024-09-01T03:15:29.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-01T03:15:29.226 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 87061' 2024-09-01T03:15:29.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-01T03:15:29.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-01T03:15:30.227 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-01T03:15:30.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:15:30.228 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:15:30.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:15:30.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:15:30.228 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:15:30.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:15:30.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 87038 87059 87060 87061 2024-09-01T03:15:30.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-01T03:15:30.342 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:30.343 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:15:30.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:15:30.345 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:15:30.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:15:30.346 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-01T03:15:30.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:15:30.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:15:30.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:15:30.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:15:30.370 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:30.368+0000 7f19f7a64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:30.372 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:30.370+0000 7f19f7a64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:30.373 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:30.372+0000 7f19f7a64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:30.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:15:30.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:15:30.767 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:15:30.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:15:30.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:15:30.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:30.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:15:30.768 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:15:30.768 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:30.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:31.192 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1995305368,v1:127.0.0.1:6803/1995305368] [v2:127.0.0.1:6804/1995305368,v1:127.0.0.1:6805/1995305368] exists,up 4b839a65-e33c-4a01-bf61-599111c9c6ac 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-01T03:15:31.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:31.197 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-01T03:15:31.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:15:31.199 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:15:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:15:31.200 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-01T03:15:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:15:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:15:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:15:31.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:15:31.220 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:31.218+0000 7fec9b80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:31.221 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:31.220+0000 7fec9b80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:31.222 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:31.221+0000 7fec9b80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:31.611 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:15:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:15:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:15:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:15:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:15:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:31.740 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:15:31.741 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:15:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:15:31.741 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:31.721+0000 7f19f7a64640 -1 Falling back to public interface 2024-09-01T03:15:32.019 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/558364149,v1:127.0.0.1:6811/558364149] [v2:127.0.0.1:6812/558364149,v1:127.0.0.1:6813/558364149] exists,up f7fc88ec-09ea-4356-8af5-140662ab416b 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:15:32.023 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:32.024 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:15:32.025 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-01T03:15:32.026 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:15:32.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:15:32.046 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.044+0000 7f361de24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:32.048 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.046+0000 7f361de24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:32.049 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.047+0000 7f361de24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:32.312 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.311+0000 7fec9b80b640 -1 Falling back to public interface 2024-09-01T03:15:32.420 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:15:32.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:15:32.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:15:32.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:15:32.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:15:32.421 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:32.422 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:15:32.422 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:15:32.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:32.423 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:15:32.872 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3300469933,v1:127.0.0.1:6819/3300469933] [v2:127.0.0.1:6820/3300469933,v1:127.0.0.1:6821/3300469933] exists,up 024208b2-e5b9-44b7-87c7-b91ae59b9da0 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-01T03:15:32.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:15:32.877 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:15:32.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-01T03:15:32.879 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-09-01T03:15:32.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-01T03:15:32.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=cbf2ec26-4620-42b4-bbaa-0aa8695b4524 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.79672/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:15:32.880 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-01T03:15:32.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-01T03:15:32.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:15:32.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:15:32.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:15:32.900 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.898+0000 7fdac4c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:32.901 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.900+0000 7fdac4c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:32.903 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:32.901+0000 7fdac4c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:33.145 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:33.144+0000 7f361de24640 -1 Falling back to public interface 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:33.291 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:15:33.292 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:15:33.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:33.292 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:15:33.707 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3680603446,v1:127.0.0.1:6827/3680603446] [v2:127.0.0.1:6828/3680603446,v1:127.0.0.1:6829/3680603446] exists,up 8783d08d-aa7c-428c-a39b-7e5e0ea7e890 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:15:33.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:15:33.710 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:15:33.710 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:15:33.792 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:15:33.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:15:33.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:15:33.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:15:33.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:15:33.793 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:15:34.002 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:34.001+0000 7fdac4c0b640 -1 Falling back to public interface 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:2 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:3' 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:15:34.192 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:15:35.322 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:35.320+0000 7f19f7a64640 -1 osd.0 26 log_to_monitors true 2024-09-01T03:15:36.409 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:36.407+0000 7fec9b80b640 -1 osd.1 26 log_to_monitors true 2024-09-01T03:15:37.791 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:37.788+0000 7f361de24640 -1 osd.2 25 log_to_monitors true 2024-09-01T03:15:38.057 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:38.055+0000 7fdac4c0b640 -1 osd.3 26 log_to_monitors true 2024-09-01T03:15:39.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051587 2024-09-01T03:15:39.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051587 2024-09-01T03:15:39.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587' 2024-09-01T03:15:39.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:15:39.356 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:15:39.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-01T03:15:39.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-01T03:15:39.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-09-01T03:15:39.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:15:39.478 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:15:39.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-01T03:15:39.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-01T03:15:39.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659' 2024-09-01T03:15:39.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:15:39.681 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T03:15:39.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789955 2024-09-01T03:15:39.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789955 2024-09-01T03:15:39.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659 3-158913789955' 2024-09-01T03:15:39.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:15:39.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051587 2024-09-01T03:15:39.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:15:39.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:15:39.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051587 2024-09-01T03:15:39.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:15:39.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051587 2024-09-01T03:15:39.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-09-01T03:15:39.800 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 124554051587 2024-09-01T03:15:39.800 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:15:40.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051587 -lt 124554051587 2024-09-01T03:15:40.309 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:15:40.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953475 2024-09-01T03:15:40.309 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:15:40.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:15:40.311 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953475 2024-09-01T03:15:40.311 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:15:40.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-01T03:15:40.312 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 137438953475 2024-09-01T03:15:40.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-09-01T03:15:40.312 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:15:40.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-01T03:15:40.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:15:40.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822659 2024-09-01T03:15:40.745 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:15:40.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:15:40.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822659 2024-09-01T03:15:40.746 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:15:40.747 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 154618822659 2024-09-01T03:15:40.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-01T03:15:40.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-09-01T03:15:40.748 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:15:41.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-01T03:15:41.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:15:41.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-158913789955 2024-09-01T03:15:41.204 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:15:41.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T03:15:41.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-158913789955 2024-09-01T03:15:41.206 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:15:41.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789955 2024-09-01T03:15:41.207 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.3 seq 158913789955 2024-09-01T03:15:41.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 158913789955' 2024-09-01T03:15:41.207 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T03:15:41.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789955 2024-09-01T03:15:41.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:15:41.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:15:41.630 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:15:42.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:15:42.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-01T03:15:42.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:15:42.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:15:42.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:15:43.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-01T03:15:43.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:15:43.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:15:43.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-01T03:15:43.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:15:43.074 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:15:43.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:15:43.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:15:43.182 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:15:43.182 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:15:43.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:15:43.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:15:43.184 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:15:43.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:15:43.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:15:43.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:15:43.185 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:15:43.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:15:43.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:15:43.187 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-01T03:15:43.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:15:43.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:43.205 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:15:43.205 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.79672 2024-09-01T03:15:43.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:15:43.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:15:43.206 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ec-error-rollforward 0 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:15:43.207 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:15:43.208 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:15:43.208 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:15:43.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:15:43.209 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:15:43.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:15:43.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:15:43.210 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:15:43.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:15:43.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:15:43.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:15:43.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:15:43.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:15:43.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T03:15:43.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-01T03:15:43.214 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:15:43.214 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:43.214 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.79672 2024-09-01T03:15:43.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.79672 2024-09-01T03:15:43.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:15:43.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:15:43.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T03:15:43.216 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T03:15:43.216 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T03:15:43.280 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-01T03:15:43.281 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-01T03:15:43.342 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T03:15:43.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bench 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-01T03:15:43.348 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=605f15cb-21d2-40e6-82af-769c34050fdc --auth-supported=none ' 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-01T03:15:43.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bench 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bench 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:15:43.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:15:43.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:15:43.352 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:15:43.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:15:43.352 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:15:43.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:15:43.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:15:43.353 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:15:43.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:15:43.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:15:43.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:15:43.354 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:15:43.355 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:15:43.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:15:43.356 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-01T03:15:43.357 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:15:43.357 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:43.357 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:43.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.89603 2024-09-01T03:15:43.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:15:43.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:15:43.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bench 2024-09-01T03:15:43.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:15:43.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:43.359 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:43.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.89603 2024-09-01T03:15:43.361 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:15:43.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bench 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bench/a 2024-09-01T03:15:43.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.89603/$cluster-$name.asok' 2024-09-01T03:15:43.819 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.89603/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:15:44.052 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:15:44.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:15:44.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:15:44.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:15:44.054 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.89603/ceph-mon.a.asok 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:15:44.055 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.89603/ceph-mon.a.asok config get fsid 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:44.133 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:44.134 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.89603/ceph-mon.a.asok 2024-09-01T03:15:44.134 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:15:44.134 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.89603/ceph-mon.a.asok config get mon_host 2024-09-01T03:15:44.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-01T03:15:44.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bench 2024-09-01T03:15:44.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:15:44.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:15:44.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:15:44.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bench/x 2024-09-01T03:15:44.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.89603/$cluster-$name.asok' 2024-09-01T03:15:45.058 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:15:45.059 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.89603/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:15:45.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-01T03:15:45.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bench 2024-09-01T03:15:45.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:15:45.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:15:45.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:15:45.088 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bench/0 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=605f15cb-21d2-40e6-82af-769c34050fdc --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:15:45.089 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.89603/$cluster-$name.asok' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.89603/$cluster-$name.asok' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:15:45.090 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bench/0 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ad49023-d619-495c-ad70-3a218b59cd61 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5ad49023-d619-495c-ad70-3a218b59cd61' 2024-09-01T03:15:45.091 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 5ad49023-d619-495c-ad70-3a218b59cd61 2024-09-01T03:15:45.092 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:15:45.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBh3NNmg/UxBhAAE17H/OtDtycOjr5M+x/f+A== 2024-09-01T03:15:45.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBh3NNmg/UxBhAAE17H/OtDtycOjr5M+x/f+A=="}' 2024-09-01T03:15:45.106 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ad49023-d619-495c-ad70-3a218b59cd61 -i td/osd-bench/0/new.json 2024-09-01T03:15:45.378 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:15:45.387 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bench/0/new.json 2024-09-01T03:15:45.388 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=605f15cb-21d2-40e6-82af-769c34050fdc --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.89603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBh3NNmg/UxBhAAE17H/OtDtycOjr5M+x/f+A== --osd-uuid 5ad49023-d619-495c-ad70-3a218b59cd61 2024-09-01T03:15:45.417 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:45.415+0000 7f5a8ed32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:45.447 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:45.445+0000 7f5a8ed32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:45.448 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:45.446+0000 7f5a8ed32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:45.448 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:45.447+0000 7f5a8ed32640 -1 bdev(0x556ca9cdbc00 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:15:45.448 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:45.447+0000 7f5a8ed32640 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-01T03:15:51.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-01T03:15:51.002 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:15:51.003 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:15:51.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:15:51.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:15:51.528 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:15:51.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:15:51.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=605f15cb-21d2-40e6-82af-769c34050fdc --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.89603/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:15:51.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:15:51.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:15:51.529 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:15:51.553 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:51.552+0000 7f98ffe1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:51.561 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:51.560+0000 7f98ffe1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:51.562 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:51.561+0000 7f98ffe1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:51.959 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:15:51.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:15:51.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:51.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:52.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:52.652 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:52.651+0000 7f98ffe1d640 -1 Falling back to public interface 2024-09-01T03:15:53.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:53.338 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:53.338 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:15:53.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:15:53.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:53.339 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:53.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:54.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:54.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:54.762 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:15:54.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:15:54.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:54.763 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:55.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:55.716 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:15:55.715+0000 7f98ffe1d640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:15:56.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:56.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:56.154 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:15:56.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:15:56.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:56.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:56.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:57.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:57.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:57.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:15:57.771 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:15:57.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:57.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:58.158 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:15:59.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:15:59.159 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:15:59.160 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:15:59.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:15:59.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:15:59.160 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:15:59.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:00.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:00.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:00.626 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:16:00.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:16:00.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:00.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:01.036 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:02.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:02.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:02.038 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:16:02.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:16:02.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:02.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:02.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:03.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:03.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:03.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:16:03.463 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:16:03.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:03.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:03.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:04.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:04.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:04.855 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:16:04.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:16:04.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:04.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:05.271 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:06.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:06.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:06.273 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:16:06.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:16:06.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:06.273 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:06.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:07.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:07.729 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:07.729 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:16:07.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:16:07.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:07.730 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:08.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:09.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:09.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:09.551 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:16:09.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:16:09.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:09.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:10.319 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:11.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:11.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:11.321 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:16:11.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:16:11.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:11.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:12.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:13.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:13.397 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:13.398 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:16:13.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:16:13.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:13.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:14.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:15.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:15.317 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:15.318 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:16:15.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:16:15.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:15.318 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:16.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:17.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:17.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:17.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:16:17.480 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:16:17.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:17.481 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:18.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:16:19.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:16:19.564 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:16:19.565 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:16:19.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:16:19.565 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:16:19.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:16:19.984 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1336259667,v1:127.0.0.1:6803/1336259667] [v2:127.0.0.1:6804/1336259667,v1:127.0.0.1:6805/1336259667] exists,up 5ad49023-d619-495c-ad70-3a218b59cd61 2024-09-01T03:16:19.984 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:16:19.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:16:19.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:16:19.985 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-01T03:16:19.985 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-01T03:16:19.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-01T03:16:19.999 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-01T03:16:19.999 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-01T03:16:20.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-01T03:16:20.013 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-01T03:16:20.013 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-01T03:16:20.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-01T03:16:20.028 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-01T03:16:20.028 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-01T03:16:20.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_max_block_size 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T03:16:20.043 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-01T03:16:20.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T03:16:20.135 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T03:16:20.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-01T03:16:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T03:16:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-01T03:16:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-01T03:16:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-01T03:16:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-01T03:16:20.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T03:16:20.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=bench_small_size_max_iops 2024-09-01T03:16:20.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T03:16:20.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T03:16:20.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-01T03:16:20.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T03:16:20.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T03:16:20.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-01T03:16:20.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-01T03:16:20.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-01T03:16:20.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-01T03:16:20.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-01T03:16:20.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-01T03:16:20.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-01T03:17:08.811 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:17:08.811 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_written": 1073741824, 2024-09-01T03:17:08.811 INFO:tasks.workunit.client.0.smithi079.stdout: "blocksize": 4194304, 2024-09-01T03:17:08.811 INFO:tasks.workunit.client.0.smithi079.stdout: "elapsed_sec": 24.173365798999999, 2024-09-01T03:17:08.811 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_per_sec": 44418383.146480098, 2024-09-01T03:17:08.812 INFO:tasks.workunit.client.0.smithi079.stdout: "iops": 10.590167795772576 2024-09-01T03:17:08.812 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:17:08.820 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_written": 10, 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout: "blocksize": 14456, 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout: "elapsed_sec": 0.050195581000000003, 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_per_sec": 199.22072423068477, 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout: "iops": 0.013781179041967679 2024-09-01T03:17:09.154 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:17:09.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout:{ 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_written": 1, 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout: "blocksize": 14456, 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout: "elapsed_sec": 0.034731665000000002, 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout: "bytes_per_sec": 28.792169911808141, 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout: "iops": 0.0019917107022556823 2024-09-01T03:17:09.405 INFO:tasks.workunit.client.0.smithi079.stdout:} 2024-09-01T03:17:09.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:17:09.415 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:17:09.520 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:17:09.520 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:17:09.521 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:17:09.521 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:17:09.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:17:09.522 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:17:09.523 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:17:09.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:17:09.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:17:09.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:17:09.524 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:17:09.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:17:09.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:17:09.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-01T03:17:09.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:17:09.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:09.552 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:17:09.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.89603 2024-09-01T03:17:09.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bench 0 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:17:09.554 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:17:09.555 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:17:09.555 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:17:09.556 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:17:09.556 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:17:09.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:17:09.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:17:09.557 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:17:09.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:17:09.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:17:09.558 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:17:09.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:17:09.559 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:17:09.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-01T03:17:09.560 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-01T03:17:09.561 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:17:09.561 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:09.561 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.89603 2024-09-01T03:17:09.561 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.89603 2024-09-01T03:17:09.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:17:09.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:17:09.563 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-01T03:17:09.563 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-01T03:17:09.563 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T03:17:09.627 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-01T03:17:09.627 DEBUG:teuthology.orchestra.run.smithi079:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-01T03:17:09.689 INFO:tasks.workunit.client.0.smithi079.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-01T03:17:09.695 INFO:tasks.workunit.client.0.smithi079.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-01T03:17:09.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T03:17:09.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-01T03:17:09.695 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bluefs-volume-ops 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-01T03:17:09.696 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_bluestore2 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:TEST_bluestore_expand' 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:09.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:17:09.699 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:17:09.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:17:09.700 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:17:09.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:17:09.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:17:09.701 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:17:09.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:17:09.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:17:09.702 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:17:09.702 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:17:09.703 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:17:09.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:17:09.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-01T03:17:09.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:17:09.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:09.705 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:09.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.91822 2024-09-01T03:17:09.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:17:09.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:17:09.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-01T03:17:09.707 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:17:09.707 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:09.707 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:09.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.91822 2024-09-01T03:17:09.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:17:09.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-01T03:17:09.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-01T03:17:09.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-01T03:17:09.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-01T03:17:09.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-01T03:17:09.711 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-01T03:17:09.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none ' 2024-09-01T03:17:09.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-01T03:17:09.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:17:10.172 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:17:10.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:17:10.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:17:10.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:17:10.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:17:10.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:17:10.349 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:17:10.349 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.91822/ceph-mon.a.asok 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:17:10.350 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.91822/ceph-mon.a.asok config get fsid 2024-09-01T03:17:10.429 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:17:10.429 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:17:10.429 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.91822/ceph-mon.a.asok 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:17:10.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.91822/ceph-mon.a.asok config get mon_host 2024-09-01T03:17:10.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-01T03:17:10.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:10.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:17:10.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:17:10.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:17:10.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-01T03:17:10.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:11.344 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:17:11.345 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:17:11.345 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:17:11.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:11.376 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:17:11.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:17:11.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:17:11.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:17:11.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:17:11.378 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:17:11.378 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:17:11.378 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:17:11.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6f546482-bd72-4bfb-aad8-467e7b87061b' 2024-09-01T03:17:11.379 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:17:11.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC33NNm/zRJFxAA0e8c7l3GqSMFLdg1yh97Mw== 2024-09-01T03:17:11.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC33NNm/zRJFxAA0e8c7l3GqSMFLdg1yh97Mw=="}' 2024-09-01T03:17:11.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6f546482-bd72-4bfb-aad8-467e7b87061b -i td/osd-bluefs-volume-ops/0/new.json 2024-09-01T03:17:11.673 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:17:11.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-01T03:17:11.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC33NNm/zRJFxAA0e8c7l3GqSMFLdg1yh97Mw== --osd-uuid 6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:17:11.711 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:11.710+0000 7f5f8f812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:11.741 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:11.740+0000 7f5f8f812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:11.742 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:11.741+0000 7f5f8f812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:11.743 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:11.741+0000 7f5f8f812640 -1 bdev(0x560c74e51c00 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:17:11.743 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:11.741+0000 7f5f8f812640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-01T03:17:21.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-01T03:17:21.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:17:21.646 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:17:21.646 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:17:21.647 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:17:22.173 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:17:22.174 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:17:22.174 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:17:22.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:17:22.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:17:22.175 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:17:22.200 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:22.198+0000 7fd6b963b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:22.212 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:22.210+0000 7fd6b963b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:22.213 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:22.211+0000 7fd6b963b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:22.617 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:17:22.618 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:22.619 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:23.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:24.027 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:24.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:24.028 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:17:24.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:17:24.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:24.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:24.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:24.561 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:24.560+0000 7fd6b963b640 -1 Falling back to public interface 2024-09-01T03:17:25.436 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:25.437 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:17:25.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:25.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:17:25.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:25.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:25.861 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:26.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:26.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:26.863 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:17:26.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:17:26.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:26.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:27.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:28.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:28.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:28.281 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:17:28.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:17:28.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:28.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:28.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:29.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:29.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:29.707 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:17:29.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:17:29.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:29.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:29.865 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:29.864+0000 7fd6b963b640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:17:30.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:31.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:31.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:31.165 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:17:31.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:17:31.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:31.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:31.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:32.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:32.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:32.576 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:17:32.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:17:32.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:32.576 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:32.964 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:33.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:33.966 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:33.966 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:17:33.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:17:33.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:33.967 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:34.361 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:35.362 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:35.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:35.363 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:17:35.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:17:35.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:35.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:35.784 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:36.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:36.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:36.786 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:17:36.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:17:36.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:36.786 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:37.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:38.200 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:38.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:38.201 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:17:38.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:17:38.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:38.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:38.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:39.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:39.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:39.625 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:17:39.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:17:39.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:39.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:40.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:41.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:41.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:41.054 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:17:41.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:17:41.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:41.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:41.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:42.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:42.691 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:42.691 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:17:42.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:17:42.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:42.692 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:43.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:44.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:44.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:44.394 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:17:44.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:17:44.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:44.395 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:45.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:46.151 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:46.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:46.152 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:17:46.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:17:46.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:46.153 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:47.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:48.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:48.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:48.077 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:17:48.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:17:48.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:48.078 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:48.745 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:49.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:49.746 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:49.747 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T03:17:49.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T03:17:49.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:49.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:50.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:17:51.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:17:51.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:17:51.211 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-09-01T03:17:51.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T03:17:51.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:17:51.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:17:51.634 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3432788684,v1:127.0.0.1:6803/3432788684] [v2:127.0.0.1:6804/3432788684,v1:127.0.0.1:6805/3432788684] exists,up 6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:17:51.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:17:51.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:17:51.635 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:17:51.635 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:17:51.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=93291 2024-09-01T03:17:51.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-01T03:17:51.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:17:51.639 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:17:51.640 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:17:51.641 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:17:51.642 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:17:51.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:17:51.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:17:51.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-01T03:17:51.643 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:17:51.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:17:51.643 INFO:tasks.workunit.client.0.smithi079.stdout:add osd1 60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:17:51.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 60417cd2-a911-4530-9d04-7f7b3d4631ae' 2024-09-01T03:17:51.644 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:17:51.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDf3NNmEu0JJxAAZ/zH3TmKrnGri5aWBu0XwA== 2024-09-01T03:17:51.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDf3NNmEu0JJxAAZ/zH3TmKrnGri5aWBu0XwA=="}' 2024-09-01T03:17:51.656 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 60417cd2-a911-4530-9d04-7f7b3d4631ae -i td/osd-bluefs-volume-ops/1/new.json 2024-09-01T03:17:52.208 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:17:52.219 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-01T03:17:52.220 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDf3NNmEu0JJxAAZ/zH3TmKrnGri5aWBu0XwA== --osd-uuid 60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:17:52.249 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:52.247+0000 7fe23ea2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:52.250 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:52.249+0000 7fe23ea2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:52.251 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:52.250+0000 7fe23ea2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:17:52.252 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:52.250+0000 7fe23ea2e640 -1 bdev(0x55d24b11fc00 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-09-01T03:17:52.252 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:17:52.250+0000 7fe23ea2e640 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-01T03:18:01.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-01T03:18:01.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:18:01.303 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd1 key to auth repository 2024-09-01T03:18:01.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-01T03:18:01.303 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:18:01.834 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:18:01.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-01T03:18:01.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:18:01.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:18:01.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:18:01.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:18:01.860 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:01.858+0000 7ff5cbdbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:01.881 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:01.880+0000 7ff5cbdbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:01.882 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:01.881+0000 7ff5cbdbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:02.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-01T03:18:02.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:18:02.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:18:02.295 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:18:02.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:18:02.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:02.296 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:18:02.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:18:02.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:02.297 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:02.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:03.681 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:03.682 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:18:03.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:03.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:18:03.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:03.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:03.987 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:03.986+0000 7ff5cbdbf640 -1 Falling back to public interface 2024-09-01T03:18:04.068 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:05.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:05.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:05.070 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:18:05.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:18:05.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:05.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:05.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:06.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:06.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:06.501 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:18:06.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:18:06.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:06.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:06.933 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:07.934 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:07.935 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:18:07.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:07.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:18:07.935 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:07.936 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:08.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:08.918 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:08.917+0000 7ff5cbdbf640 -1 osd.1 0 log_to_monitors true 2024-09-01T03:18:09.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:09.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:09.354 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:18:09.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:18:09.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:09.354 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:09.785 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:10.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:10.787 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:10.787 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:18:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:18:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:10.788 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:11.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:12.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:12.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:12.198 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:18:12.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:18:12.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:12.199 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:12.643 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:13.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:13.644 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:13.645 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:18:13.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:18:13.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:13.645 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:14.051 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:15.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:15.053 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:15.053 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:18:15.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:18:15.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:15.054 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:15.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:16.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:16.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:16.511 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:18:16.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:18:16.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:16.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:17.037 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:18.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:18.038 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:18.038 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:18:18.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:18:18.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:18.039 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:18.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:19.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:19.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:19.431 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:18:19.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:18:19.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:19.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:19.862 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:20.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:20.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:20.864 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:18:20.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:18:20.864 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:20.865 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:21.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:22.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:22.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:22.336 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:18:22.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:18:22.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:22.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:23.003 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:24.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:24.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:24.005 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:18:24.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:18:24.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:24.005 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:25.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:26.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:26.092 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:26.093 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:18:26.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:18:26.093 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:26.094 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:26.885 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:27.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:27.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:27.887 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:18:27.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:18:27.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:27.888 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:28.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:29.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:29.548 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:29.548 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T03:18:29.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T03:18:29.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:29.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:30.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:31.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:31.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:31.251 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-09-01T03:18:31.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T03:18:31.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:31.252 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:31.698 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:32.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:32.700 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:32.700 INFO:tasks.workunit.client.0.smithi079.stdout:20 2024-09-01T03:18:32.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-01T03:18:32.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:32.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:18:33.124 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/464519711,v1:127.0.0.1:6811/464519711] [v2:127.0.0.1:6812/464519711,v1:127.0.0.1:6813/464519711] exists,up 60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:18:33.124 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:18:33.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:18:33.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:18:33.125 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-01T03:18:33.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=95846 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:18:33.129 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:18:33.130 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-01T03:18:33.131 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:18:33.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:18:33.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c93287fd-c50b-4217-a32f-e3115cb0b464' 2024-09-01T03:18:33.132 INFO:tasks.workunit.client.0.smithi079.stdout:add osd2 c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:18:33.133 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:18:33.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAJ3dNmR9OrCBAAPvzD7Q6j0OBeCYuusC9JkA== 2024-09-01T03:18:33.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAJ3dNmR9OrCBAAPvzD7Q6j0OBeCYuusC9JkA=="}' 2024-09-01T03:18:33.147 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c93287fd-c50b-4217-a32f-e3115cb0b464 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-01T03:18:33.638 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:18:33.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-01T03:18:33.651 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJ3dNmR9OrCBAAPvzD7Q6j0OBeCYuusC9JkA== --osd-uuid c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:18:33.679 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:33.677+0000 7f60e6c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:33.690 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:33.688+0000 7f60e6c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:33.691 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:33.689+0000 7f60e6c1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:33.691 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:33.690+0000 7f60e6c1a640 -1 bdev(0x55c9232afc00 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-09-01T03:18:33.691 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:33.690+0000 7f60e6c1a640 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-01T03:18:42.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-01T03:18:42.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:18:42.957 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd2 key to auth repository 2024-09-01T03:18:42.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-01T03:18:42.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:18:43.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-01T03:18:43.488 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:18:43.488 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:18:43.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:18:43.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:18:43.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:18:43.514 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:43.512+0000 7ffabe22f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:43.546 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:43.544+0000 7ffabe22f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:43.547 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:43.546+0000 7ffabe22f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:18:43.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-01T03:18:43.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:18:43.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:18:43.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:18:43.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:18:43.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:43.952 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:18:43.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:18:43.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:43.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:44.368 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:45.370 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:45.370 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:18:45.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:45.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:18:45.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:45.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:45.632 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:45.630+0000 7ffabe22f640 -1 Falling back to public interface 2024-09-01T03:18:45.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:46.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:46.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:46.797 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:18:46.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:18:46.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:46.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:47.183 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:48.184 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:48.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:48.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:18:48.185 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:18:48.185 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:48.186 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:48.570 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:49.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:49.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:49.571 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:18:49.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:18:49.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:49.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:49.975 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:50.693 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:18:50.691+0000 7ffabe22f640 -1 osd.2 0 log_to_monitors true 2024-09-01T03:18:50.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:50.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:50.976 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:18:50.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:18:50.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:50.977 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:51.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:52.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:52.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:52.493 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:18:52.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:18:52.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:52.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:52.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:53.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:53.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:53.913 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:18:53.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:18:53.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:53.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:54.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:55.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:55.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:55.327 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:18:55.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:18:55.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:55.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:55.835 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:56.836 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:56.837 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:56.837 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:18:56.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:18:56.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:56.838 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:57.229 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:58.230 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:58.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:58.231 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:18:58.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:18:58.231 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:58.232 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:18:58.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:18:59.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:18:59.623 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:18:59.624 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:18:59.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:18:59.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:18:59.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:00.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:01.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:01.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:01.035 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:19:01.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:19:01.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:01.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:01.523 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:02.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:02.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:02.525 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:19:02.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:19:02.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:02.526 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:03.176 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:04.177 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:04.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:04.178 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:19:04.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:19:04.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:04.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:04.904 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:05.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:05.906 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:05.906 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:19:05.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:19:05.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:05.907 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:06.968 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:07.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:07.969 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:07.969 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:19:07.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:19:07.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:07.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:08.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:09.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:09.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:09.605 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:19:09.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:19:09.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:09.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:10.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:11.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:11.463 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:11.463 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T03:19:11.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T03:19:11.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:11.464 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:12.384 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:13.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:13.385 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:13.386 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-09-01T03:19:13.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T03:19:13.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:13.386 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:19:13.840 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3643834695,v1:127.0.0.1:6819/3643834695] [v2:127.0.0.1:6820/3643834695,v1:127.0.0.1:6821/3643834695] exists,up c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:19:13.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:19:13.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:19:13.848 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:19:13.848 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=98425 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:19:13.849 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-01T03:19:13.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:19:13.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:19:13.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:19:13.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:19:13.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-01T03:19:13.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-01T03:19:13.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:19:13.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:19:13.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:19:13.851 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:19:13.851 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:19:13.851 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:19:13.852 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:19:13.852 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:19:13.852 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:19:13.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:19:13.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-01T03:19:13.855 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:19:13.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:19:13.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 aaf1b283-1c9c-427e-932b-d45e2f1720de' 2024-09-01T03:19:13.855 INFO:tasks.workunit.client.0.smithi079.stdout:add osd3 aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:19:13.856 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:19:13.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAx3dNmX7VaMxAA4p9aZpYYq9RZ5T/SIiraBA== 2024-09-01T03:19:13.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAx3dNmX7VaMxAA4p9aZpYYq9RZ5T/SIiraBA=="}' 2024-09-01T03:19:13.863 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aaf1b283-1c9c-427e-932b-d45e2f1720de -i td/osd-bluefs-volume-ops/3/new.json 2024-09-01T03:19:14.380 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:19:14.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-01T03:19:14.392 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAx3dNmX7VaMxAA4p9aZpYYq9RZ5T/SIiraBA== --osd-uuid aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:19:14.421 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:14.419+0000 7fdd6a414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:19:14.422 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:14.421+0000 7fdd6a414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:19:14.423 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:14.422+0000 7fdd6a414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:19:14.424 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:14.422+0000 7fdd6a414640 -1 bdev(0x56169b82fc00 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-09-01T03:19:14.424 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:14.422+0000 7fdd6a414640 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-01T03:19:24.489 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-01T03:19:24.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:19:24.490 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd3 key to auth repository 2024-09-01T03:19:24.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-01T03:19:24.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:19:25.029 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-09-01T03:19:25.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-01T03:19:25.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:19:25.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:19:25.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:19:25.031 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:19:25.055 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:25.054+0000 7f3eaec0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:19:25.077 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:25.075+0000 7f3eaec0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:19:25.078 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:25.076+0000 7f3eaec0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:19:25.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-01T03:19:25.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:19:25.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T03:19:25.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:19:25.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:19:25.499 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:25.500 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:19:25.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:19:25.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:25.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:25.916 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:26.917 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:26.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:26.918 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:19:26.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:19:26.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:26.918 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:26.923 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:26.922+0000 7f3eaec0e640 -1 Falling back to public interface 2024-09-01T03:19:27.334 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:28.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:28.335 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:28.335 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:19:28.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:19:28.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:28.336 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:28.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:29.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:29.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:29.762 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:19:29.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:19:29.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:29.762 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:30.178 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:31.179 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:31.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:31.180 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:19:31.180 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:19:31.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:31.181 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:31.648 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:31.723 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:19:31.721+0000 7f3eaec0e640 -1 osd.3 0 log_to_monitors true 2024-09-01T03:19:32.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:32.649 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:32.650 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:19:32.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:19:32.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:32.650 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:33.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:34.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:34.072 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:34.072 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:19:34.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:19:34.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:34.073 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:34.613 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:35.614 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:35.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:35.615 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:19:35.615 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:19:35.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:35.616 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:36.013 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:37.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:37.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:37.015 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:19:37.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:19:37.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:37.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:37.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:38.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:38.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:38.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-01T03:19:38.410 INFO:tasks.workunit.client.0.smithi079.stdout:9 2024-09-01T03:19:38.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:38.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:38.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:39.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:39.931 INFO:tasks.workunit.client.0.smithi079.stdout:10 2024-09-01T03:19:39.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:39.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-01T03:19:39.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:39.932 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:40.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:41.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:41.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:41.323 INFO:tasks.workunit.client.0.smithi079.stdout:11 2024-09-01T03:19:41.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-01T03:19:41.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:41.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:41.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:42.727 INFO:tasks.workunit.client.0.smithi079.stdout:12 2024-09-01T03:19:42.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-01T03:19:42.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:42.728 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:43.163 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:44.164 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:44.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:44.165 INFO:tasks.workunit.client.0.smithi079.stdout:13 2024-09-01T03:19:44.165 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-01T03:19:44.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:44.166 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:45.060 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:46.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:46.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:46.062 INFO:tasks.workunit.client.0.smithi079.stdout:14 2024-09-01T03:19:46.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-01T03:19:46.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:46.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:46.758 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:47.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:47.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:47.760 INFO:tasks.workunit.client.0.smithi079.stdout:15 2024-09-01T03:19:47.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-01T03:19:47.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:47.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:48.469 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:49.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:49.471 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:49.471 INFO:tasks.workunit.client.0.smithi079.stdout:16 2024-09-01T03:19:49.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-01T03:19:49.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:49.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:49.879 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:50.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:50.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:50.881 INFO:tasks.workunit.client.0.smithi079.stdout:17 2024-09-01T03:19:50.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-01T03:19:50.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:50.882 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:51.624 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:52.625 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:52.626 INFO:tasks.workunit.client.0.smithi079.stdout:18 2024-09-01T03:19:52.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:52.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-01T03:19:52.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:52.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:53.272 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:19:54.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:19:54.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:19:54.274 INFO:tasks.workunit.client.0.smithi079.stdout:19 2024-09-01T03:19:54.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-01T03:19:54.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:19:54.275 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:19:54.657 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/507463928,v1:127.0.0.1:6827/507463928] [v2:127.0.0.1:6828/507463928,v1:127.0.0.1:6829/507463928] exists,up aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:19:54.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:19:54.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:19:54.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:19:54.658 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-01T03:19:54.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=100979 2024-09-01T03:19:54.658 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-01T03:19:59.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-01T03:19:59.660 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-01T03:20:00.142 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-09-01T03:20:00.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T03:20:01.156 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-01T03:20:01.218 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-09-01T03:20:01.218 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-01T03:20:01.218 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_102344 2024-09-01T03:20:01.218 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:20:01.218 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-09-01T03:20:02.218 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 50 34 0.132798 0.132812 0.284565 0.356798 2024-09-01T03:20:03.218 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 113 97 0.189434 0.246094 0.301248 0.317204 2024-09-01T03:20:04.219 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 167 151 0.196593 0.210938 0.268751 0.305186 2024-09-01T03:20:05.218 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 226 210 0.205054 0.230469 0.267939 0.297277 2024-09-01T03:20:06.219 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 281 265 0.207005 0.214844 0.287104 0.291522 2024-09-01T03:20:07.219 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 345 329 0.214166 0.25 0.239004 0.286375 2024-09-01T03:20:08.219 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 402 386 0.215375 0.222656 0.23536 0.284137 2024-09-01T03:20:09.219 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 466 450 0.219698 0.25 0.276672 0.27939 2024-09-01T03:20:10.219 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 530 514 0.223061 0.25 0.26903 0.278183 2024-09-01T03:20:11.219 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 586 570 0.222627 0.21875 0.21006 0.275942 2024-09-01T03:20:12.219 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 650 634 0.225113 0.25 0.269746 0.275361 2024-09-01T03:20:13.220 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 706 690 0.22458 0.21875 0.268478 0.273884 2024-09-01T03:20:14.220 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 770 754 0.226533 0.25 0.276985 0.274096 2024-09-01T03:20:15.220 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 826 810 0.225975 0.21875 0.260918 0.273369 2024-09-01T03:20:16.220 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 882 866 0.225491 0.21875 0.277088 0.273412 2024-09-01T03:20:17.220 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 946 930 0.227021 0.25 0.277429 0.272438 2024-09-01T03:20:18.220 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 1002 986 0.226533 0.21875 0.284933 0.273015 2024-09-01T03:20:19.220 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 1058 1042 0.226099 0.21875 0.259526 0.274501 2024-09-01T03:20:20.221 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 1107 1091 0.224271 0.191406 0.356323 0.275948 2024-09-01T03:20:21.221 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T03:20:21.220397+0000 min lat: 0.134662 max lat: 0.474538 avg lat: 0.276928 2024-09-01T03:20:21.221 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:20:21.221 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 1164 1148 0.224189 0.222656 0.269926 0.276928 2024-09-01T03:20:22.221 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 1228 1212 0.225416 0.25 0.235046 0.27627 2024-09-01T03:20:23.221 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 1292 1276 0.226533 0.25 0.272793 0.275015 2024-09-01T03:20:24.221 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 1347 1331 0.226023 0.214844 0.243601 0.274352 2024-09-01T03:20:25.221 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 1411 1395 0.227021 0.25 0.252423 0.273557 2024-09-01T03:20:26.221 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 1475 1459 0.227938 0.25 0.243644 0.272653 2024-09-01T03:20:27.222 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 1539 1523 0.228785 0.25 0.249425 0.271561 2024-09-01T03:20:28.222 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 1603 1587 0.22957 0.25 0.26067 0.27073 2024-09-01T03:20:29.222 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 1644 1628 0.22709 0.160156 0.483932 0.271515 2024-09-01T03:20:30.222 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 1692 1676 0.225724 0.1875 0.268726 0.275532 2024-09-01T03:20:31.222 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 1756 1740 0.226532 0.25 0.268378 0.27484 2024-09-01T03:20:32.222 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1714 2024-09-01T03:20:32.222 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 1756 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.227347 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0290149 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.25 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.132812 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 58 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 7.42781 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 64 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 34 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.274435 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0570662 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.810872 2024-09-01T03:20:32.223 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.134662 2024-09-01T03:20:32.351 INFO:tasks.workunit.client.0.smithi079.stdout:after bench 2024-09-01T03:20:32.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-01T03:20:32.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 93291 2024-09-01T03:20:32.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-01T03:20:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 93291 2024-09-01T03:20:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (93291) - No such process 2024-09-01T03:20:33.353 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-01T03:20:33.977 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:20:33.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 95846 2024-09-01T03:20:33.989 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-01T03:20:34.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 95846 2024-09-01T03:20:34.990 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (95846) - No such process 2024-09-01T03:20:34.991 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-01T03:20:35.558 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T03:20:35.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 98425 2024-09-01T03:20:35.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-01T03:20:36.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 98425 2024-09-01T03:20:36.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (98425) - No such process 2024-09-01T03:20:36.571 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-01T03:20:36.984 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T03:20:36.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 100979 2024-09-01T03:20:36.996 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-01T03:20:37.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 100979 2024-09-01T03:20:37.997 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (100979) - No such process 2024-09-01T03:20:37.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-01T03:20:38.494 INFO:tasks.workunit.client.0.smithi079.stderr:osd.3 is already down. 2024-09-01T03:20:38.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:20:40.813 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:20:40.824 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:20:42.889 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:20:42.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:20:45.212 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:20:45.222 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:20:47.535 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:20:47.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-01T03:20:47.562 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-01T03:20:47.585 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:20:48.693 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-01T03:20:48.693 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-01T03:20:48.693 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x100000000 : using 0x80568000(2.0 GiB) 2024-09-01T03:20:48.693 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-09-01T03:20:48.693 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-09-01T03:20:48.703 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-09-01T03:20:48.703 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-09-01T03:20:48.711 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-09-01T03:20:48.712 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-01T03:20:48.837 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4294967296 2024-09-01T03:20:53.875 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-01T03:20:53.876 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-01T03:20:53.901 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:20:55.011 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-01T03:20:55.011 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-01T03:20:55.011 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x101000000 : using 0x81557000(2.0 GiB) 2024-09-01T03:20:55.011 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-09-01T03:20:55.011 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-09-01T03:20:55.042 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-09-01T03:20:55.042 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-09-01T03:20:55.061 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-09-01T03:20:55.061 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-01T03:20:55.228 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4311744512 2024-09-01T03:21:00.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-01T03:21:00.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-01T03:21:00.652 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:01.763 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-01T03:21:01.763 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-01T03:21:01.763 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x100020000 : using 0x8056c000(2.0 GiB) 2024-09-01T03:21:01.763 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-09-01T03:21:01.763 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-09-01T03:21:01.772 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-09-01T03:21:01.772 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-09-01T03:21:01.780 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-09-01T03:21:01.780 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-01T03:21:01.905 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4295098368 2024-09-01T03:21:06.844 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-01T03:21:06.845 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-01T03:21:06.867 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:07.980 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-01T03:21:07.980 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-01T03:21:07.980 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0xfff00000 : using 0x80498000(2.0 GiB) 2024-09-01T03:21:07.980 INFO:tasks.workunit.client.0.smithi079.stdout:Expanding DB/WAL... 2024-09-01T03:21:07.980 INFO:tasks.workunit.client.0.smithi079.stdout:0 : expanding to 0x536870912 2024-09-01T03:21:07.986 INFO:tasks.workunit.client.0.smithi079.stdout:0 : size label updated to 536870912 2024-09-01T03:21:07.986 INFO:tasks.workunit.client.0.smithi079.stdout:1 : expanding to 0x1073741824 2024-09-01T03:21:07.994 INFO:tasks.workunit.client.0.smithi079.stdout:1 : size label updated to 1073741824 2024-09-01T03:21:07.994 INFO:tasks.workunit.client.0.smithi079.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-01T03:21:08.136 INFO:tasks.workunit.client.0.smithi079.stdout:2 : size label updated to 4293918720 2024-09-01T03:21:13.242 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:21:15.559 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:15.569 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:21:17.879 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:17.887 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:21:20.203 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:20.213 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:21:22.527 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:22.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-01T03:21:22.561 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:24.586 INFO:tasks.workunit.client.0.smithi079.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-01T03:21:24.586 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-01T03:21:24.587 INFO:tasks.workunit.client.0.smithi079.stdout:2 : device size 0x100000000 : using 0x568000(5.4 MiB) 2024-09-01T03:21:24.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-01T03:21:24.611 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:27.184 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-01T03:21:27.189 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:21:29.004 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:29.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-01T03:21:29.037 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:31.855 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-01T03:21:31.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:21:33.680 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:33.686 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-01T03:21:33.706 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:36.526 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-01T03:21:36.526 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-01T03:21:36.530 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:21:37.844 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:37.850 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-01T03:21:37.871 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:38.663 INFO:tasks.workunit.client.0.smithi079.stderr:Migrate to WAL device isn't supported. 2024-09-01T03:21:38.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-01T03:21:38.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:21:40.977 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:40.985 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-01T03:21:41.008 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:43.575 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-01T03:21:43.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:21:45.391 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:45.400 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-01T03:21:45.422 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:21:47.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:21:49.301 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:21:49.312 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:21:49.313 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:21:49.314 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:21:49.315 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:21:49.316 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:21:49.340 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:49.338+0000 7f3f1c815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:21:49.342 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:49.341+0000 7f3f1c815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:21:49.343 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:49.342+0000 7f3f1c815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:21:49.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:21:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:21:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:21:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:21:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:21:49.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:49.725 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:21:49.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:21:49.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:49.726 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:21:50.152 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:21:50.933 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:50.932+0000 7f3f1c815640 -1 Falling back to public interface 2024-09-01T03:21:51.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:21:51.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:51.154 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:21:51.154 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:21:51.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:51.155 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:21:51.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:21:52.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:21:52.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:52.550 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:21:52.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:21:52.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:52.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:21:52.970 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:21:53.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:21:53.972 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:53.972 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:21:53.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:21:53.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:53.973 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:21:54.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:21:54.870 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:54.867+0000 7f3f1c815640 -1 osd.0 27 log_to_monitors true 2024-09-01T03:21:55.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:21:55.374 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:21:55.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:55.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:21:55.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:55.375 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:21:55.789 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/2631079597,v1:127.0.0.1:6803/2631079597] [v2:127.0.0.1:6804/2631079597,v1:127.0.0.1:6805/2631079597] exists,up 6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:21:55.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:21:55.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:21:55.790 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:21:55.790 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:21:55.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=114488 2024-09-01T03:21:55.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-01T03:21:55.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:21:55.793 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:21:55.794 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:21:55.795 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:21:55.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:21:55.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:21:55.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:21:55.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-01T03:21:55.796 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:21:55.796 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:21:55.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:21:55.797 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-01T03:21:55.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:21:55.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:21:55.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:21:55.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:21:55.819 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:55.817+0000 7fb7060fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:21:55.823 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:55.821+0000 7fb7060fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:21:55.824 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:55.822+0000 7fb7060fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:21:56.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:21:56.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:21:56.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:21:56.254 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:21:56.254 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:21:56.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:21:56.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:56.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:21:56.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:56.255 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:21:56.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:21:57.429 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:21:57.427+0000 7fb7060fe640 -1 Falling back to public interface 2024-09-01T03:21:57.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:21:57.706 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:57.707 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:21:57.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:21:57.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:57.707 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:21:58.098 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:21:59.099 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:21:59.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:21:59.100 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:21:59.100 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:21:59.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:21:59.101 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:21:59.533 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:00.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:00.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:00.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:22:00.535 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:22:00.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:00.535 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:22:00.949 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:01.748 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:01.746+0000 7fb7060fe640 -1 osd.1 29 log_to_monitors true 2024-09-01T03:22:01.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:01.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:01.951 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:22:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:22:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:01.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:22:02.402 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6810/860170565,v1:127.0.0.1:6811/860170565] [v2:127.0.0.1:6812/860170565,v1:127.0.0.1:6813/860170565] exists,up 60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:22:02.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:22:02.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:22:02.403 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:22:02.403 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-01T03:22:02.406 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=115461 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:22:02.407 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:22:02.408 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-01T03:22:02.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:22:02.410 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:22:02.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:22:02.411 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-01T03:22:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:22:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:22:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:22:02.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:22:02.431 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:02.430+0000 7f5cf040d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:22:02.433 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:02.432+0000 7f5cf040d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:22:02.435 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:02.433+0000 7f5cf040d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:22:03.075 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:22:03.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:22:03.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:22:03.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:22:03.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:22:03.076 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:03.076 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:22:03.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:22:03.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:03.077 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:22:03.294 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:03.293+0000 7f5cf040d640 -1 Falling back to public interface 2024-09-01T03:22:03.572 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:04.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:04.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:04.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:22:04.574 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:22:04.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:04.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:22:04.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:05.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:05.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:05.959 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:22:05.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:22:05.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:05.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:22:06.344 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:06.568 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:06.566+0000 7f5cf040d640 -1 osd.2 33 log_to_monitors true 2024-09-01T03:22:07.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:07.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:07.346 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:22:07.346 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:22:07.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:07.347 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:22:07.945 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 34 last_clean_interval [15,33) [v2:127.0.0.1:6818/376000319,v1:127.0.0.1:6819/376000319] [v2:127.0.0.1:6820/376000319,v1:127.0.0.1:6821/376000319] exists,up c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:22:07.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:22:07.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:22:07.946 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:22:07.946 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=116434 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:22:07.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:22:07.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-01T03:22:07.952 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-09-01T03:22:07.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-01T03:22:07.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:22:07.954 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-01T03:22:07.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-01T03:22:07.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:22:07.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:22:07.954 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:22:07.974 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:07.973+0000 7f466ab34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:22:07.976 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:07.975+0000 7f466ab34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:22:07.978 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:07.976+0000 7f466ab34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:22:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-01T03:22:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:22:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T03:22:08.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:22:08.428 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:22:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:22:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:22:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:08.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:22:08.851 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:09.316 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:09.315+0000 7f466ab34640 -1 Falling back to public interface 2024-09-01T03:22:09.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:09.853 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:09.853 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:22:09.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:22:09.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:09.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:22:10.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:11.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:11.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:11.263 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:22:11.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:22:11.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:11.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:22:11.680 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:12.682 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:12.747 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:12.747 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:22:12.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:22:12.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:12.748 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:22:13.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:22:13.662 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:22:13.661+0000 7f466ab34640 -1 osd.3 35 log_to_monitors true 2024-09-01T03:22:14.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:22:14.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:22:14.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:22:14.109 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:22:14.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:22:14.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:22:14.689 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 49 up_thru 34 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/3555374134,v1:127.0.0.1:6827/3555374134] [v2:127.0.0.1:6828/3555374134,v1:127.0.0.1:6829/3555374134] exists,up aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:22:14.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:22:14.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:22:14.690 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:22:14.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-01T03:22:14.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=117377 2024-09-01T03:22:14.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-01T03:22:14.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:22:14.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:22:14.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:22:14.714 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:22:14.715 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:22:14.715 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:22:14.715 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:22:14.715 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:22:14.715 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:22:14.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:22:14.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:22:14.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:22:14.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:22:14.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:22:14.800 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr:2 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr:3' 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:22:15.587 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:22:15.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757255 2024-09-01T03:22:15.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757255 2024-09-01T03:22:15.750 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255' 2024-09-01T03:22:15.751 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:22:15.751 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:22:15.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659141 2024-09-01T03:22:15.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659141 2024-09-01T03:22:15.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255 1-176093659141' 2024-09-01T03:22:15.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:22:15.899 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:22:16.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528324 2024-09-01T03:22:16.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528324 2024-09-01T03:22:16.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324' 2024-09-01T03:22:16.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:22:16.015 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T03:22:16.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397507 2024-09-01T03:22:16.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397507 2024-09-01T03:22:16.125 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324 3-210453397507' 2024-09-01T03:22:16.126 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:22:16.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-163208757255 2024-09-01T03:22:16.126 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:22:16.127 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:22:16.127 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-163208757255 2024-09-01T03:22:16.127 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:22:16.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757255 2024-09-01T03:22:16.128 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 163208757255' 2024-09-01T03:22:16.129 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 163208757255 2024-09-01T03:22:16.129 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:22:16.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757253 -lt 163208757255 2024-09-01T03:22:16.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:22:17.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:22:17.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:22:17.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757255 -lt 163208757255 2024-09-01T03:22:17.976 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:22:17.977 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659141 2024-09-01T03:22:17.977 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:22:17.978 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:22:17.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659141 2024-09-01T03:22:17.978 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:22:17.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659141 2024-09-01T03:22:17.979 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-09-01T03:22:17.980 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 176093659141 2024-09-01T03:22:17.980 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:22:18.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659142 -lt 176093659141 2024-09-01T03:22:18.394 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:22:18.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-193273528324 2024-09-01T03:22:18.395 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:22:18.396 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:22:18.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-193273528324 2024-09-01T03:22:18.397 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:22:18.398 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528324 2024-09-01T03:22:18.398 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 193273528324 2024-09-01T03:22:18.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 193273528324' 2024-09-01T03:22:18.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:22:18.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528325 -lt 193273528324 2024-09-01T03:22:18.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:22:18.855 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-210453397507 2024-09-01T03:22:18.855 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:22:18.855 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T03:22:18.856 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-210453397507 2024-09-01T03:22:18.857 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:22:18.857 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.3 seq 210453397507 2024-09-01T03:22:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397507 2024-09-01T03:22:18.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2024-09-01T03:22:18.858 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T03:22:19.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-01T03:22:19.264 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:22:19.264 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:22:19.264 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:22:19.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:22:19.798 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:22:20.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-01T03:22:20.211 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:22:20.211 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:22:20.211 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:22:20.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-01T03:22:20.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:22:20.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:22:20.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-01T03:22:20.817 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-09-01T03:22:20.817 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-01T03:22:20.817 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_118718 2024-09-01T03:22:20.818 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:22:20.818 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-09-01T03:22:21.817 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 74 58 0.226544 0.226562 0.151229 0.24406 2024-09-01T03:22:22.818 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 133 117 0.228491 0.230469 0.125856 0.250477 2024-09-01T03:22:23.818 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 201 185 0.240857 0.265625 0.260121 0.251763 2024-09-01T03:22:24.818 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 259 243 0.237279 0.226562 0.208266 0.251051 2024-09-01T03:22:25.818 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 316 300 0.234351 0.222656 0.232144 0.254308 2024-09-01T03:22:26.818 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 374 358 0.233049 0.226562 0.129268 0.260136 2024-09-01T03:22:27.818 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 427 411 0.229328 0.207031 0.263224 0.267279 2024-09-01T03:22:28.818 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 493 477 0.232883 0.257812 0.260183 0.264499 2024-09-01T03:22:29.818 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 550 534 0.231743 0.222656 0.367027 0.266777 2024-09-01T03:22:30.818 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 609 593 0.231612 0.230469 0.205852 0.266116 2024-09-01T03:22:31.818 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 668 652 0.231507 0.230469 0.299205 0.264911 2024-09-01T03:22:32.819 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 721 705 0.229464 0.207031 0.208867 0.268778 2024-09-01T03:22:33.819 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 782 766 0.23014 0.238281 0.318719 0.269121 2024-09-01T03:22:34.819 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 853 837 0.233509 0.277344 0.175668 0.266454 2024-09-01T03:22:35.819 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 905 889 0.231482 0.203125 0.197639 0.266369 2024-09-01T03:22:36.819 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 974 958 0.233857 0.269531 0.287524 0.264886 2024-09-01T03:22:37.819 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 1031 1015 0.233197 0.222656 0.254295 0.265217 2024-09-01T03:22:38.819 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 1117 1101 0.238902 0.335938 0.181091 0.259769 2024-09-01T03:22:39.820 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 1190 1174 0.241335 0.285156 0.171938 0.257518 2024-09-01T03:22:40.820 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T03:22:40.819498+0000 min lat: 0.108731 max lat: 0.47218 avg lat: 0.25421 2024-09-01T03:22:40.820 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:22:40.820 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 1268 1252 0.2445 0.304688 0.209551 0.25421 2024-09-01T03:22:41.820 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 1348 1332 0.247736 0.3125 0.217675 0.250336 2024-09-01T03:22:42.820 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 1418 1402 0.248903 0.273438 0.301083 0.250114 2024-09-01T03:22:43.820 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 1492 1476 0.250647 0.289062 0.254412 0.248302 2024-09-01T03:22:44.820 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 1572 1556 0.253223 0.3125 0.186801 0.245933 2024-09-01T03:22:45.820 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 1643 1627 0.254186 0.277344 0.227362 0.244839 2024-09-01T03:22:46.821 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 1723 1707 0.256428 0.3125 0.221596 0.242819 2024-09-01T03:22:47.821 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 1802 1786 0.258359 0.308594 0.238198 0.241104 2024-09-01T03:22:48.821 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 1868 1852 0.258338 0.257812 0.192483 0.241037 2024-09-01T03:22:49.821 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 1945 1929 0.259801 0.300781 0.18574 0.239348 2024-09-01T03:22:50.821 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 2024 2008 0.261426 0.308594 0.201352 0.238248 2024-09-01T03:22:51.821 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1003 2024-09-01T03:22:51.821 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 2024 2024-09-01T03:22:51.821 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.262664 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0390481 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.335938 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.203125 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 67 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 9.99632 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 86 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 52 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.237693 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0609263 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.47218 2024-09-01T03:22:51.822 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.108731 2024-09-01T03:22:51.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 114488 2024-09-01T03:22:51.897 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-01T03:22:52.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 114488 2024-09-01T03:22:52.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (114488) - No such process 2024-09-01T03:22:52.898 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-01T03:22:53.479 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:22:53.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 115461 2024-09-01T03:22:53.490 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-01T03:22:54.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 115461 2024-09-01T03:22:54.491 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (115461) - No such process 2024-09-01T03:22:54.492 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-01T03:22:54.938 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T03:22:54.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 116434 2024-09-01T03:22:54.950 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-01T03:22:55.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 116434 2024-09-01T03:22:55.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (116434) - No such process 2024-09-01T03:22:55.951 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-01T03:22:56.352 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T03:22:56.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 117377 2024-09-01T03:22:56.363 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-01T03:22:57.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 117377 2024-09-01T03:22:57.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (117377) - No such process 2024-09-01T03:22:57.365 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-01T03:22:57.947 INFO:tasks.workunit.client.0.smithi079.stderr:osd.3 is already down. 2024-09-01T03:22:57.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:22:59.514 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:22:59.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-01T03:22:59.780 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records in 2024-09-01T03:22:59.781 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records out 2024-09-01T03:22:59.781 INFO:tasks.workunit.client.0.smithi079.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.254596 s, 2.1 GB/s 2024-09-01T03:22:59.781 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-01T03:22:59.801 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:05.114 INFO:tasks.workunit.client.0.smithi079.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-01T03:23:05.117 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:23:07.434 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:07.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:23:09.260 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:09.267 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-01T03:23:09.771 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-09-01T03:23:09.771 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-09-01T03:23:09.771 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.50226 s, 2.1 GB/s 2024-09-01T03:23:09.771 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-01T03:23:09.789 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:18.590 INFO:tasks.workunit.client.0.smithi079.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-01T03:23:18.593 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-01T03:23:18.615 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:19.679 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:19.676+0000 7f5879fdc980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:20.442 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:20.439+0000 7f5879fdc980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:21.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:23:21.996 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:21.994+0000 7f442edf8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:22.758 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:22.757+0000 7f442edf8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:23.755 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:23.765 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:23:25.076 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:25.083 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-01T03:23:25.104 INFO:tasks.workunit.client.0.smithi079.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-01T03:23:25.104 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:26.390 INFO:tasks.workunit.client.0.smithi079.stdout:DB device added 2024-09-01T03:23:26.393 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-01T03:23:26.417 INFO:tasks.workunit.client.0.smithi079.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-01T03:23:26.417 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:27.214 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:27.213+0000 7fd8c780f980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:27.727 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:27.725+0000 7fd8c780f980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:27.783 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:27.781+0000 7fd8c780f980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:28.477 INFO:tasks.workunit.client.0.smithi079.stdout:WAL device added 2024-09-01T03:23:28.480 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-01T03:23:28.503 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:29.569 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:29.568+0000 7fe4fa71e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:30.333 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:30.331+0000 7fe4fa71e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:31.580 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:23:32.132 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:32.130+0000 7ffa147f8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:32.898 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:32.896+0000 7ffa147f8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:33.898 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:33.908 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:23:35.723 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:35.731 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-01T03:23:35.753 INFO:tasks.workunit.client.0.smithi079.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-01T03:23:35.753 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:37.801 INFO:tasks.workunit.client.0.smithi079.stdout:WAL device added 2024-09-01T03:23:37.804 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-01T03:23:37.827 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:23:40.888 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-01T03:23:40.894 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:23:42.711 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:23:42.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:23:42.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:23:42.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:23:42.723 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:23:42.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:23:42.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:23:42.726 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:23:42.727 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:23:42.749 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:42.748+0000 7f876021c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:42.751 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:42.750+0000 7f876021c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:42.752 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:42.751+0000 7f876021c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:43.103 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:23:43.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:23:43.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:23:43.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:23:43.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:23:43.104 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:43.104 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:23:43.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:23:43.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:43.105 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:23:43.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:44.356 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:44.355+0000 7f876021c640 -1 Falling back to public interface 2024-09-01T03:23:44.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:44.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:44.546 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:23:44.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:23:44.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:44.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:23:44.953 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:45.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:45.955 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:45.955 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:23:45.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:23:45.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:45.956 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:23:46.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:47.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:47.381 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:47.382 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:23:47.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:23:47.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:47.382 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:23:47.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:48.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:48.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:48.799 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:23:48.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:23:48.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:48.800 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:23:49.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:49.242 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:49.240+0000 7f876021c640 -1 osd.0 51 log_to_monitors true 2024-09-01T03:23:50.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:50.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:50.192 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:23:50.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:23:50.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:50.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:23:50.598 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 62 up_thru 62 down_at 52 last_clean_interval [38,51) [v2:127.0.0.1:6802/1514225226,v1:127.0.0.1:6803/1514225226] [v2:127.0.0.1:6804/1514225226,v1:127.0.0.1:6805/1514225226] exists,up 6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:23:50.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:23:50.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:23:50.599 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:23:50.599 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:23:50.602 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=124870 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-01T03:23:50.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:23:50.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-01T03:23:50.605 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:23:50.606 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:23:50.628 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:50.626+0000 7f7540f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:50.630 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:50.628+0000 7f7540f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:50.631 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:50.629+0000 7f7540f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:51.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:23:51.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:23:51.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:23:51.017 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:23:51.018 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:23:51.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:23:51.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:51.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:23:51.018 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:51.019 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:23:51.475 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:52.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:52.476 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:52.476 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:23:52.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:23:52.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:52.477 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:23:52.718 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:52.717+0000 7f7540f45640 -1 Falling back to public interface 2024-09-01T03:23:52.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:53.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:53.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:53.912 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:23:53.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:23:53.913 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:53.914 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:23:54.321 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:54.867 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:54.866+0000 7f7540f45640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:55.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:55.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:55.324 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:23:55.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:23:55.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:55.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:23:55.629 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:55.628+0000 7f7540f45640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:55.759 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:56.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:56.760 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:56.761 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:23:56.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:23:56.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:56.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:23:56.898 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:56.897+0000 7f7540f45640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:57.191 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:23:57.661 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:57.660+0000 7f7540f45640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:23:57.980 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:57.979+0000 7f7540f45640 -1 osd.1 54 log_to_monitors true 2024-09-01T03:23:58.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:23:58.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:58.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:23:58.193 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:23:58.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:58.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:23:58.703 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/3876143093,v1:127.0.0.1:6811/3876143093] [v2:127.0.0.1:6812/3876143093,v1:127.0.0.1:6813/3876143093] exists,up 60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:23:58.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:23:58.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:23:58.704 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:23:58.704 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=125874 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:23:58.708 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:23:58.709 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-01T03:23:58.710 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:23:58.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:23:58.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:23:58.711 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-01T03:23:58.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:23:58.711 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:23:58.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:23:58.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:23:58.733 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:58.732+0000 7effeed43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:58.827 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:58.826+0000 7effeed43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:58.828 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:23:58.827+0000 7effeed43640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:23:59.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:23:59.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:23:59.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:23:59.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:23:59.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:23:59.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:23:59.140 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:23:59.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:23:59.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:23:59.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:23:59.566 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:00.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:00.567 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:00.568 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:24:00.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:24:00.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:00.568 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:24:00.907 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:00.905+0000 7effeed43640 -1 Falling back to public interface 2024-09-01T03:24:00.957 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:01.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:01.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:01.959 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:24:01.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:24:01.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:01.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:24:02.377 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:03.313 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:03.311+0000 7effeed43640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:24:03.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:03.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:03.379 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:24:03.379 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:24:03.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:03.380 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:24:03.805 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:04.079 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:04.078+0000 7effeed43640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:24:04.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:04.807 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:04.807 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:24:04.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:24:04.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:04.808 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:24:05.090 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:05.088+0000 7effeed43640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:24:05.201 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:05.867 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:05.865+0000 7effeed43640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:24:06.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:06.202 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:06.202 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:24:06.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:24:06.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:06.203 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:24:06.590 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:06.752 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:06.750+0000 7effeed43640 -1 osd.2 57 log_to_monitors true 2024-09-01T03:24:07.591 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:07.591 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:24:07.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:07.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:24:07.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:07.592 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:24:08.106 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 68 up_thru 55 down_at 58 last_clean_interval [45,57) [v2:127.0.0.1:6818/824610145,v1:127.0.0.1:6819/824610145] [v2:127.0.0.1:6820/824610145,v1:127.0.0.1:6821/824610145] exists,up c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:24:08.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:24:08.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:24:08.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:24:08.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-01T03:24:08.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=126894 2024-09-01T03:24:08.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-01T03:24:08.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:24:08.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-01T03:24:08.114 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-09-01T03:24:08.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:24:08.116 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-01T03:24:08.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-01T03:24:08.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:24:08.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:24:08.116 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:24:08.137 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:08.135+0000 7facc0389640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:24:08.138 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:08.137+0000 7facc0389640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:24:08.140 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:08.138+0000 7facc0389640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:08.524 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:24:08.525 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:24:08.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:08.525 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:24:09.084 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:10.085 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:10.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:10.086 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:24:10.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:24:10.086 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:10.087 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:24:10.229 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:10.228+0000 7facc0389640 -1 Falling back to public interface 2024-09-01T03:24:10.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:11.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:11.513 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:11.513 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:24:11.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:24:11.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:11.514 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:24:11.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:12.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:12.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:12.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:24:12.901 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:24:12.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:12.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:24:13.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:14.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:14.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:14.326 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:24:14.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:24:14.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:14.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:24:14.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:24:15.252 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:24:15.251+0000 7facc0389640 -1 osd.3 59 log_to_monitors true 2024-09-01T03:24:15.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:24:15.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:24:15.721 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:24:15.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:24:15.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:24:15.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:24:16.209 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 73 up_thru 58 down_at 60 last_clean_interval [49,59) [v2:127.0.0.1:6826/3394502213,v1:127.0.0.1:6827/3394502213] [v2:127.0.0.1:6828/3394502213,v1:127.0.0.1:6829/3394502213] exists,up aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:24:16.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:24:16.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:24:16.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:24:16.210 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-01T03:24:16.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=127905 2024-09-01T03:24:16.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-01T03:24:16.382 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-09-01T03:24:16.382 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-01T03:24:16.382 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_128888 2024-09-01T03:24:16.382 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:24:16.382 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-09-01T03:24:17.381 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 26 10 0.0390577 0.0390625 0.301809 0.281901 2024-09-01T03:24:18.382 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 80 64 0.124985 0.210938 0.354021 0.463146 2024-09-01T03:24:19.382 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 132 116 0.151025 0.203125 0.320954 0.392359 2024-09-01T03:24:20.382 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 182 166 0.162091 0.195312 0.218971 0.363858 2024-09-01T03:24:21.382 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 243 227 0.177324 0.238281 0.391758 0.341956 2024-09-01T03:24:22.382 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 303 287 0.186827 0.234375 0.267115 0.330388 2024-09-01T03:24:23.382 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 357 341 0.190268 0.210938 0.280007 0.320229 2024-09-01T03:24:24.382 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 414 398 0.194314 0.222656 0.302421 0.316669 2024-09-01T03:24:25.383 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 471 455 0.19746 0.222656 0.184057 0.311015 2024-09-01T03:24:26.383 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 530 514 0.200757 0.230469 0.26754 0.30787 2024-09-01T03:24:27.383 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 582 566 0.20097 0.203125 0.175481 0.30743 2024-09-01T03:24:28.383 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 635 619 0.201473 0.207031 0.202413 0.304994 2024-09-01T03:24:29.383 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 696 680 0.204302 0.238281 0.283846 0.301647 2024-09-01T03:24:30.383 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 755 739 0.206169 0.230469 0.16766 0.301307 2024-09-01T03:24:31.383 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 812 796 0.207266 0.222656 0.270095 0.299654 2024-09-01T03:24:32.383 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 863 847 0.206762 0.199219 0.247636 0.30053 2024-09-01T03:24:33.383 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 914 898 0.206317 0.199219 0.340847 0.29955 2024-09-01T03:24:34.384 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 964 948 0.205704 0.195312 0.356705 0.299906 2024-09-01T03:24:35.384 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 1022 1006 0.206801 0.226562 0.31453 0.299973 2024-09-01T03:24:36.384 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T03:24:36.383707+0000 min lat: 0.143421 max lat: 1.1719 avg lat: 0.300774 2024-09-01T03:24:36.384 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:24:36.384 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 1071 1055 0.20603 0.191406 0.293447 0.300774 2024-09-01T03:24:37.384 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 1119 1103 0.205146 0.1875 0.385142 0.301758 2024-09-01T03:24:38.384 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 1171 1155 0.205053 0.203125 0.340999 0.303106 2024-09-01T03:24:39.384 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 1220 1204 0.204459 0.191406 0.326657 0.303964 2024-09-01T03:24:40.385 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 1274 1258 0.204728 0.210938 0.193084 0.303456 2024-09-01T03:24:41.385 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 1332 1316 0.2056 0.226562 0.284071 0.301995 2024-09-01T03:24:42.385 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 1376 1360 0.204302 0.171875 0.27726 0.30247 2024-09-01T03:24:43.385 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 1432 1416 0.204836 0.21875 0.291424 0.303569 2024-09-01T03:24:44.385 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 1484 1468 0.204774 0.203125 0.294011 0.303909 2024-09-01T03:24:45.385 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 1538 1522 0.204986 0.210938 0.17614 0.303163 2024-09-01T03:24:46.385 INFO:tasks.workunit.client.0.smithi079.stdout: 30 13 1591 1578 0.205444 0.21875 0.223378 0.303078 2024-09-01T03:24:47.385 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.121 2024-09-01T03:24:47.385 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 1591 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.206329 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0353964 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.238281 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.0390625 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 52 2024-09-01T03:24:47.386 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 9.06148 2024-09-01T03:24:47.387 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 61 2024-09-01T03:24:47.387 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 10 2024-09-01T03:24:47.387 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.30237 2024-09-01T03:24:47.387 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0930727 2024-09-01T03:24:47.387 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 1.1719 2024-09-01T03:24:47.387 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.143421 2024-09-01T03:24:47.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 124870 2024-09-01T03:24:47.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-01T03:24:48.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 124870 2024-09-01T03:24:48.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (124870) - No such process 2024-09-01T03:24:48.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-01T03:24:48.992 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:24:49.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 125874 2024-09-01T03:24:49.004 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-01T03:24:50.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 125874 2024-09-01T03:24:50.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (125874) - No such process 2024-09-01T03:24:50.006 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-01T03:24:50.459 INFO:tasks.workunit.client.0.smithi079.stderr:osd.1 is already down. 2024-09-01T03:24:50.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 126894 2024-09-01T03:24:50.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-01T03:24:51.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 126894 2024-09-01T03:24:51.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (126894) - No such process 2024-09-01T03:24:51.472 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-01T03:24:51.916 INFO:tasks.workunit.client.0.smithi079.stderr:osd.2 is already down. 2024-09-01T03:24:51.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 127905 2024-09-01T03:24:51.928 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-01T03:24:52.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 127905 2024-09-01T03:24:52.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (127905) - No such process 2024-09-01T03:24:52.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-01T03:24:53.417 INFO:tasks.workunit.client.0.smithi079.stderr:osd.3 is already down. 2024-09-01T03:24:53.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:24:55.734 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:24:55.744 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-01T03:24:56.236 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-09-01T03:24:56.236 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-09-01T03:24:56.236 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.490956 s, 2.2 GB/s 2024-09-01T03:24:56.236 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-01T03:24:56.256 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:25:05.827 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-01T03:25:05.828 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-01T03:25:05.831 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:25:08.142 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:08.150 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-01T03:25:08.413 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records in 2024-09-01T03:25:08.413 INFO:tasks.workunit.client.0.smithi079.stderr:512+0 records out 2024-09-01T03:25:08.413 INFO:tasks.workunit.client.0.smithi079.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.261774 s, 2.1 GB/s 2024-09-01T03:25:08.414 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-01T03:25:08.433 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:25:14.499 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-01T03:25:14.500 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-01T03:25:14.503 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-01T03:25:14.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-01T03:25:16.869 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:16.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:25:17.462 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:17.460+0000 7fb8babcf980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:18.217 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:18.216+0000 7fb8babcf980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:19.190 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:19.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-01T03:25:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-09-01T03:25:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-09-01T03:25:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.505975 s, 2.1 GB/s 2024-09-01T03:25:19.705 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-01T03:25:19.724 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:25:20.780 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:20.778+0000 7f952611e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:21.562 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:21.560+0000 7f952611e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:27.997 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:27.996+0000 7f952611e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:29.292 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-01T03:25:29.292 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-01T03:25:29.292 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-01T03:25:29.296 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-01T03:25:29.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:25:29.950 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:29.949+0000 7f788ed1e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:30.472 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:30.470+0000 7f788ed1e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:31.209 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:31.216 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-01T03:25:31.238 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:25:32.029 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:32.028+0000 7fb0f3c23980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:32.543 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:32.541+0000 7fb0f3c23980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:33.296 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-01T03:25:33.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-01T03:25:33.405 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-01T03:25:34.711 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:34.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-01T03:25:36.768 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:36.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-01T03:25:37.274 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-09-01T03:25:37.274 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-09-01T03:25:37.274 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.499519 s, 2.1 GB/s 2024-09-01T03:25:37.274 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-01T03:25:37.293 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:25:38.091 INFO:tasks.workunit.client.0.smithi079.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-01T03:25:38.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-01T03:25:38.091 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:25:38.641 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:38.640+0000 7ff968814980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:39.429 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:39.427+0000 7ff968814980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:40.401 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:40.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-01T03:25:47.631 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-09-01T03:25:47.632 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-09-01T03:25:47.632 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.10703 s, 151 MB/s 2024-09-01T03:25:47.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-01T03:25:47.654 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:25:48.705 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:48.704+0000 7f11997f8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:49.470 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:49.468+0000 7f11997f8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:49.540 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:49.539+0000 7f11997f8980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:50.715 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-01T03:25:50.715 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-01T03:25:50.715 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-01T03:25:50.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-01T03:25:51.275 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:51.274+0000 7f2f4391e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:51.799 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:25:51.797+0000 7f2f4391e980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:25:52.535 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:25:52.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-01T03:25:53.042 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records in 2024-09-01T03:25:53.042 INFO:tasks.workunit.client.0.smithi079.stderr:1024+0 records out 2024-09-01T03:25:53.042 INFO:tasks.workunit.client.0.smithi079.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.49724 s, 2.2 GB/s 2024-09-01T03:25:53.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-01T03:25:53.061 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:26:02.115 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-01T03:26:02.116 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-01T03:26:02.116 INFO:tasks.workunit.client.0.smithi079.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-01T03:26:02.121 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-01T03:26:02.685 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:02.684+0000 7fe1f880b980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:03.200 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:03.198+0000 7fe1f880b980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:03.948 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:26:03.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:26:03.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:26:03.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:26:03.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:26:03.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:26:03.960 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:26:03.961 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:26:03.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:26:03.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:26:03.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:26:03.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:26:03.962 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:26:03.962 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:26:03.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:26:03.963 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-01T03:26:03.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:26:03.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:26:03.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:26:03.963 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:26:03.987 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:03.986+0000 7f89971a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:03.993 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:03.988+0000 7f89971a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:03.993 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:03.989+0000 7f89971a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:04.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:26:04.349 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:26:04.350 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:04.351 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:26:04.772 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:05.082 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:05.081+0000 7f89971a8640 -1 Falling back to public interface 2024-09-01T03:26:05.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:05.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:05.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:26:05.774 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:26:05.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:05.775 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:26:06.209 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:07.210 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:07.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:07.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:26:07.211 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:26:07.211 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:07.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:26:07.602 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:08.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:08.603 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:08.603 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:26:08.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:26:08.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:08.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:26:09.028 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:10.029 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:10.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:10.030 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:26:10.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:26:10.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:10.030 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:26:10.080 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:10.078+0000 7f89971a8640 -1 osd.0 75 log_to_monitors true 2024-09-01T03:26:10.621 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 86 up_thru 73 down_at 76 last_clean_interval [62,75) [v2:127.0.0.1:6802/1616168021,v1:127.0.0.1:6803/1616168021] [v2:127.0.0.1:6804/1616168021,v1:127.0.0.1:6805/1616168021] exists,up 6f546482-bd72-4bfb-aad8-467e7b87061b 2024-09-01T03:26:10.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:26:10.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:26:10.622 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:26:10.622 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=135062 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-01T03:26:10.626 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:26:10.627 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-01T03:26:10.628 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.1 2024-09-01T03:26:10.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:26:10.629 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-01T03:26:10.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-01T03:26:10.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:26:10.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:26:10.630 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:26:10.651 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:10.649+0000 7fb63f820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:10.703 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:10.701+0000 7fb63f820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:10.704 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:10.702+0000 7fb63f820640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:11.033 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-01T03:26:11.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:26:11.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-01T03:26:11.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:26:11.034 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:26:11.034 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:26:11.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:11.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:26:11.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:11.035 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:26:11.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:12.055 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:12.054+0000 7fb63f820640 -1 Falling back to public interface 2024-09-01T03:26:12.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:12.445 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:12.445 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:26:12.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:26:12.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:12.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:26:12.869 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:13.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:13.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:13.871 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:26:13.871 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:26:13.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:13.872 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:26:14.263 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:15.051 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:15.050+0000 7fb63f820640 -1 osd.1 78 log_to_monitors true 2024-09-01T03:26:15.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:15.264 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:15.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:26:15.265 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:26:15.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:15.265 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-01T03:26:15.769 INFO:tasks.workunit.client.0.smithi079.stdout:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 79 last_clean_interval [65,78) [v2:127.0.0.1:6810/551372069,v1:127.0.0.1:6811/551372069] [v2:127.0.0.1:6812/551372069,v1:127.0.0.1:6813/551372069] exists,up 60417cd2-a911-4530-9d04-7f7b3d4631ae 2024-09-01T03:26:15.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:26:15.769 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:26:15.770 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:26:15.770 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-01T03:26:15.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=136057 2024-09-01T03:26:15.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-01T03:26:15.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:26:15.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:26:15.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:26:15.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:26:15.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:26:15.775 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:26:15.775 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:26:15.775 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:26:15.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:15.776 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-01T03:26:15.777 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-01T03:26:15.778 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.2 2024-09-01T03:26:15.778 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:26:15.779 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-01T03:26:15.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-01T03:26:15.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:26:15.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:26:15.779 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:26:15.800 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:15.798+0000 7f9a65930640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:15.801 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:15.800+0000 7f9a65930640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:15.802 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:15.801+0000 7f9a65930640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:16.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-01T03:26:16.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:26:16.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-01T03:26:16.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:26:16.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:26:16.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:16.302 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:26:16.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:26:16.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:16.302 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:26:16.718 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:17.176 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:17.175+0000 7f9a65930640 -1 Falling back to public interface 2024-09-01T03:26:17.719 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:17.720 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:17.720 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:26:17.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:26:17.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:17.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:26:18.136 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:19.137 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:19.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:19.138 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:26:19.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:26:19.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:19.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:26:19.320 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:19.319+0000 7f9a65930640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:19.596 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:19.833 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:19.832+0000 7f9a65930640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:20.590 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:20.588+0000 7f9a65930640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:20.597 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:20.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:20.598 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:26:20.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:26:20.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:20.598 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:26:21.014 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:21.118 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:21.117+0000 7f9a65930640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:21.401 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:21.399+0000 7f9a65930640 -1 osd.2 81 log_to_monitors true 2024-09-01T03:26:22.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:22.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:22.015 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:26:22.016 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:26:22.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:22.016 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-01T03:26:22.510 INFO:tasks.workunit.client.0.smithi079.stdout:osd.2 up in weight 1 up_from 92 up_thru 92 down_at 82 last_clean_interval [68,81) [v2:127.0.0.1:6818/459114953,v1:127.0.0.1:6819/459114953] [v2:127.0.0.1:6820/459114953,v1:127.0.0.1:6821/459114953] exists,up c93287fd-c50b-4217-a32f-e3115cb0b464 2024-09-01T03:26:22.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:26:22.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:26:22.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:26:22.511 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=136983 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:26:22.515 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:26:22.516 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:26:22.517 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:26:22.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:26:22.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:26:22.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-01T03:26:22.518 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.3 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:26:22.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:26:22.540 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:22.538+0000 7faaa5aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:22.542 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:22.541+0000 7faaa5aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:22.543 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:22.542+0000 7faaa5aa5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:26:23.021 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:26:23.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:23.022 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:26:23.470 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:24.387 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:24.386+0000 7faaa5aa5640 -1 Falling back to public interface 2024-09-01T03:26:24.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:24.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:24.473 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:26:24.473 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:26:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:24.474 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:26:24.890 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:25.891 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:25.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:25.892 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:26:25.892 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:26:25.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:25.893 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:26:26.279 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:26.538 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:26.537+0000 7faaa5aa5640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:27.073 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:27.071+0000 7faaa5aa5640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:27.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:27.281 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:26:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:26:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:27.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:26:27.683 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:26:27.839 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:27.838+0000 7faaa5aa5640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:28.346 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:28.345+0000 7faaa5aa5640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-01T03:26:28.606 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:26:28.604+0000 7faaa5aa5640 -1 osd.3 83 log_to_monitors true 2024-09-01T03:26:28.684 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:26:28.684 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:26:28.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:26:28.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:26:28.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:26:28.685 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-01T03:26:29.287 INFO:tasks.workunit.client.0.smithi079.stdout:osd.3 up in weight 1 up_from 96 up_thru 82 down_at 84 last_clean_interval [73,83) [v2:127.0.0.1:6826/1975665163,v1:127.0.0.1:6827/1975665163] [v2:127.0.0.1:6828/1975665163,v1:127.0.0.1:6829/1975665163] exists,up aaf1b283-1c9c-427e-932b-d45e2f1720de 2024-09-01T03:26:29.287 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:26:29.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:26:29.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:26:29.288 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-01T03:26:29.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=137956 2024-09-01T03:26:29.288 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-01T03:26:29.433 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-09-01T03:26:29.433 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-01T03:26:29.433 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_138914 2024-09-01T03:26:29.433 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:26:29.434 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-09-01T03:26:30.433 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 60 44 0.171863 0.171875 0.500101 0.332209 2024-09-01T03:26:31.433 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 189 173 0.337856 0.503906 0.134579 0.181213 2024-09-01T03:26:32.433 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 301 285 0.371056 0.4375 0.117062 0.166356 2024-09-01T03:26:33.434 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 445 429 0.418903 0.5625 0.0959104 0.147156 2024-09-01T03:26:34.434 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 541 525 0.410111 0.375 0.143458 0.149067 2024-09-01T03:26:35.434 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 662 646 0.420528 0.472656 0.14736 0.147491 2024-09-01T03:26:36.434 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 753 737 0.41123 0.355469 0.142747 0.150806 2024-09-01T03:26:37.434 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 877 861 0.420367 0.484375 0.139615 0.147614 2024-09-01T03:26:38.434 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 985 969 0.420531 0.421875 0.116767 0.148096 2024-09-01T03:26:39.436 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 1107 1091 0.42613 0.476562 0.115053 0.146 2024-09-01T03:26:40.434 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 1212 1196 0.424675 0.410156 0.125137 0.146381 2024-09-01T03:26:41.434 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 1308 1292 0.420532 0.375 0.249887 0.147726 2024-09-01T03:26:42.434 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 1422 1406 0.422436 0.445312 0.104281 0.147379 2024-09-01T03:26:43.434 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 1539 1523 0.424904 0.457031 0.180617 0.146729 2024-09-01T03:26:44.435 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 1649 1633 0.425221 0.429688 0.153153 0.145867 2024-09-01T03:26:45.435 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 1792 1776 0.433552 0.558594 0.108902 0.143941 2024-09-01T03:26:46.435 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 1897 1881 0.432173 0.410156 0.0999046 0.143952 2024-09-01T03:26:47.435 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 1989 1973 0.428125 0.359375 0.142514 0.145399 2024-09-01T03:26:48.435 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 2089 2073 0.42615 0.390625 0.138927 0.146305 2024-09-01T03:26:49.435 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T03:26:49.435113+0000 min lat: 0.0413928 max lat: 0.906811 avg lat: 0.144904 2024-09-01T03:26:49.435 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:26:49.435 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 2218 2202 0.430035 0.503906 0.0849016 0.144904 2024-09-01T03:26:50.436 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 2349 2333 0.433919 0.511719 0.100224 0.143957 2024-09-01T03:26:51.436 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 2466 2450 0.434968 0.457031 0.112488 0.143274 2024-09-01T03:26:52.436 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 2574 2558 0.434397 0.421875 0.14312 0.143395 2024-09-01T03:26:53.436 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 2704 2688 0.437454 0.507812 0.0842999 0.142547 2024-09-01T03:26:54.436 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 2817 2801 0.43761 0.441406 0.129233 0.14247 2024-09-01T03:26:55.436 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 2929 2913 0.437604 0.4375 0.1254 0.142263 2024-09-01T03:26:56.436 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 3028 3012 0.435719 0.386719 0.110609 0.1431 2024-09-01T03:26:57.436 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 3150 3134 0.437176 0.476562 0.101679 0.142499 2024-09-01T03:26:58.436 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 3222 3206 0.431798 0.28125 0.134735 0.144469 2024-09-01T03:26:59.436 INFO:tasks.workunit.client.0.smithi079.stdout: 30 16 3342 3326 0.433029 0.46875 0.141227 0.144039 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.1484 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 3342 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.433014 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0787203 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.5625 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.171875 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 110 2024-09-01T03:27:00.437 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 20.1524 2024-09-01T03:27:00.438 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 144 2024-09-01T03:27:00.438 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 44 2024-09-01T03:27:00.438 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.144089 2024-09-01T03:27:00.438 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0544803 2024-09-01T03:27:00.438 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.906811 2024-09-01T03:27:00.438 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.0413928 2024-09-01T03:27:00.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-01T03:27:00.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:27:00.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:27:00.487 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:27:00.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:27:00.488 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:27:00.488 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:27:00.488 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:27:00.489 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:27:00.489 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:27:00.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:27:00.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:27:00.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:27:00.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:27:00.585 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:27:00.585 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr:1 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr:2 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr:3' 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:27:01.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:27:01.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187469 2024-09-01T03:27:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187469 2024-09-01T03:27:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469' 2024-09-01T03:27:01.114 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:27:01.114 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-01T03:27:01.224 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089356 2024-09-01T03:27:01.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089356 2024-09-01T03:27:01.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469 1-382252089356' 2024-09-01T03:27:01.225 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:27:01.225 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-01T03:27:01.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991242 2024-09-01T03:27:01.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991242 2024-09-01T03:27:01.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-395136991242' 2024-09-01T03:27:01.337 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:27:01.337 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-01T03:27:01.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860425 2024-09-01T03:27:01.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860425 2024-09-01T03:27:01.449 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-395136991242 3-412316860425' 2024-09-01T03:27:01.450 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:27:01.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-369367187469 2024-09-01T03:27:01.450 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:27:01.451 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:27:01.451 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-369367187469 2024-09-01T03:27:01.451 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:27:01.452 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187469 2024-09-01T03:27:01.452 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 369367187469 2024-09-01T03:27:01.453 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 369367187469' 2024-09-01T03:27:01.453 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:27:01.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187469 -lt 369367187469 2024-09-01T03:27:01.877 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:27:01.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089356 2024-09-01T03:27:01.878 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:27:01.878 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-01T03:27:01.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089356 2024-09-01T03:27:01.879 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:27:01.880 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089356 2024-09-01T03:27:01.880 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.1 seq 382252089356 2024-09-01T03:27:01.881 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-09-01T03:27:01.881 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-01T03:27:02.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089356 -lt 382252089356 2024-09-01T03:27:02.259 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:27:02.259 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-395136991242 2024-09-01T03:27:02.259 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:27:02.261 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-01T03:27:02.261 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-395136991242 2024-09-01T03:27:02.261 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:27:02.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991242 2024-09-01T03:27:02.262 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.2 seq 395136991242 2024-09-01T03:27:02.262 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 395136991242' 2024-09-01T03:27:02.263 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-01T03:27:02.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 395136991242 -lt 395136991242 2024-09-01T03:27:02.721 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:27:02.721 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-412316860425 2024-09-01T03:27:02.722 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:27:02.722 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-01T03:27:02.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-412316860425 2024-09-01T03:27:02.723 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:27:02.724 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860425 2024-09-01T03:27:02.724 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.3 seq 412316860425 2024-09-01T03:27:02.725 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 412316860425' 2024-09-01T03:27:02.725 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T03:27:03.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860423 -lt 412316860425 2024-09-01T03:27:03.132 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:27:04.133 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:27:04.134 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-01T03:27:04.557 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860425 -lt 412316860425 2024-09-01T03:27:04.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:27:04.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:27:04.558 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:27:05.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-01T03:27:05.079 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:27:05.079 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:27:05.079 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:27:05.080 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:27:05.080 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:27:05.080 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:27:05.080 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:27:05.478 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-01T03:27:05.479 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:27:05.479 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:27:05.479 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:27:05.999 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-01T03:27:06.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:27:06.000 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:27:06.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:27:06.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:27:06.000 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:27:06.107 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:27:06.107 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:27:06.108 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:27:06.108 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:27:06.109 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:27:06.110 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:27:06.110 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:27:06.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:27:06.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:27:06.111 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:27:06.111 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:27:06.112 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:27:06.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:27:06.113 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-01T03:27:06.534 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:27:06.534 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:06.534 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:06.534 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.91822 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:06.536 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:27:06.537 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:27:06.538 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:27:06.538 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:27:06.538 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:27:06.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:27:06.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:27:06.540 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:27:06.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:27:06.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:27:06.541 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:27:06.541 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:27:06.542 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:27:06.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:27:06.543 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-01T03:27:06.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:27:06.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:06.544 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:06.544 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.91822 2024-09-01T03:27:06.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:27:06.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:27:06.545 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-01T03:27:06.546 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:27:06.546 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:06.546 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:06.546 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.91822 2024-09-01T03:27:06.548 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-01T03:27:06.549 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-01T03:27:06.550 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-01T03:27:06.550 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-01T03:27:06.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none ' 2024-09-01T03:27:06.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-01T03:27:06.551 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:06.552 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:27:06.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:27:06.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:27:06.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-01T03:27:06.553 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:06.997 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:27:06.998 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:27:07.190 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:27:07.193 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:27:07.194 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:27:07.195 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:27:07.195 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:07.195 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:07.195 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.91822/ceph-mon.a.asok 2024-09-01T03:27:07.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:27:07.195 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.91822/ceph-mon.a.asok config get fsid 2024-09-01T03:27:07.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:27:07.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:27:07.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:27:07.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:27:07.274 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:27:07.274 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:27:07.275 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:27:07.275 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:27:07.275 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:27:07.275 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:07.275 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:07.275 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.91822/ceph-mon.a.asok 2024-09-01T03:27:07.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:27:07.276 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.91822/ceph-mon.a.asok config get mon_host 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-01T03:27:07.352 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:27:08.211 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:27:08.211 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:27:08.211 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:27:08.211 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:27:08.211 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:08.211 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:08.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:27:08.212 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:27:08.212 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:27:08.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:27:08.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:27:08.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:27:08.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:27:08.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:27:08.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:27:08.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-01T03:27:08.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:27:08.247 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:27:08.248 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:27:08.249 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:27:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:27:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:27:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:27:08.250 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:27:08.250 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2c6aeb72-e5f9-4471-888d-dcd73903d2a3 2024-09-01T03:27:08.251 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 2c6aeb72-e5f9-4471-888d-dcd73903d2a3 2024-09-01T03:27:08.251 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2c6aeb72-e5f9-4471-888d-dcd73903d2a3' 2024-09-01T03:27:08.251 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:27:08.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAM39NmQXvwDxAAz7QPaNMIzsqGmAe47VgBeA== 2024-09-01T03:27:08.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAM39NmQXvwDxAAz7QPaNMIzsqGmAe47VgBeA=="}' 2024-09-01T03:27:08.269 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2c6aeb72-e5f9-4471-888d-dcd73903d2a3 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-01T03:27:08.530 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:27:08.539 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-01T03:27:08.540 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAM39NmQXvwDxAAz7QPaNMIzsqGmAe47VgBeA== --osd-uuid 2c6aeb72-e5f9-4471-888d-dcd73903d2a3 2024-09-01T03:27:08.574 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:08.573+0000 7f7666bbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:27:08.583 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:08.582+0000 7f7666bbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:27:08.584 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:08.583+0000 7f7666bbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:27:08.584 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:08.583+0000 7f7666bbb640 -1 bdev(0x55f9d0828800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:27:08.585 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:08.583+0000 7f7666bbb640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-01T03:27:16.061 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-01T03:27:16.062 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:27:16.062 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:27:16.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:27:16.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:27:16.600 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:27:16.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:27:16.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:27:16.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:27:16.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:27:16.601 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:27:16.632 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:16.631+0000 7f0ae0532640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:27:16.636 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:16.634+0000 7f0ae0532640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:27:16.637 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:16.636+0000 7f0ae0532640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:27:17.040 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:27:17.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:27:17.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:27:17.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:27:17.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:27:17.041 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:17.041 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:27:17.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:27:17.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:17.042 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:17.466 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:27:18.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:27:18.467 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:18.468 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:27:18.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:27:18.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:18.468 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:18.481 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:18.480+0000 7f0ae0532640 -1 Falling back to public interface 2024-09-01T03:27:18.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:27:19.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:27:19.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:19.902 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:27:19.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:27:19.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:19.902 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:20.326 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:27:21.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:27:21.327 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:21.328 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:27:21.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:27:21.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:21.328 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:21.752 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:27:22.595 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:27:22.594+0000 7f0ae0532640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:27:22.753 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:27:22.754 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:27:22.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:22.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:27:22.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:22.754 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:23.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:27:24.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:27:24.857 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:27:24.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:24.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:27:24.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:24.858 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:26.045 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:27:27.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:27:27.047 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:27:27.047 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:27:27.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:27:27.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:27:27.048 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:27:27.457 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3298860909,v1:127.0.0.1:6803/3298860909] [v2:127.0.0.1:6804/3298860909,v1:127.0.0.1:6805/3298860909] exists,up 2c6aeb72-e5f9-4471-888d-dcd73903d2a3 2024-09-01T03:27:27.461 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:27:27.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:27:27.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:27:27.462 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:27:27.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=140813 2024-09-01T03:27:27.462 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-01T03:27:32.459 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-01T03:27:32.460 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-01T03:27:32.947 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-09-01T03:27:32.958 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T03:27:33.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-01T03:27:33.959 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-01T03:27:33.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-01T03:27:33.960 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-01T03:27:34.019 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-09-01T03:27:34.020 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-01T03:27:34.021 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_142104 2024-09-01T03:27:34.048 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:27:34.049 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-09-01T03:27:35.049 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 32 16 63.9931 64 0.981787 0.907032 2024-09-01T03:27:36.049 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 49 33 65.993 68 0.768339 0.813586 2024-09-01T03:27:37.049 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 65 49 65.3251 64 1.13604 0.932742 2024-09-01T03:27:38.049 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 81 65 64.9923 64 0.701449 0.877122 2024-09-01T03:27:39.050 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 81 65 51.9935 0 - 0.877122 2024-09-01T03:27:40.049 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 113 97 64.6587 64 0.81821 0.974786 2024-09-01T03:27:41.049 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 129 113 64.5638 64 1.00265 0.971403 2024-09-01T03:27:42.050 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 144 128 63.9923 60 1.0024 0.974644 2024-09-01T03:27:43.050 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 145 129 57.3264 4 1.01927 0.97499 2024-09-01T03:27:44.050 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 177 161 64.3921 128 0.776288 0.984101 2024-09-01T03:27:45.050 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 10.9411 2024-09-01T03:27:45.050 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 177 2024-09-01T03:27:45.050 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4194304 2024-09-01T03:27:45.050 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4194304 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 64.7103 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 35.69 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 128 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 16 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 8.92251 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 32 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 0 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.982666 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.245333 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 1.52106 2024-09-01T03:27:45.051 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.0986864 2024-09-01T03:27:45.097 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-01T03:27:55.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-01T03:27:55.099 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-01T03:27:55.233 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1060110336 2024-09-01T03:27:55.234 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-01T03:27:55.234 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-01T03:27:55.357 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=129368064 2024-09-01T03:27:55.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-01T03:27:55.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 129368064 -eq 0 2024-09-01T03:27:55.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-01T03:27:55.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 129368064 -gt 0 2024-09-01T03:27:55.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 140813 2024-09-01T03:27:55.358 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-01T03:27:56.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 140813 2024-09-01T03:27:56.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (140813) - No such process 2024-09-01T03:27:56.359 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-01T03:27:56.762 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:27:56.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-01T03:27:56.796 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:28:06.613 INFO:tasks.workunit.client.0.smithi079.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-01T03:28:06.632 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-01T03:28:06.657 INFO:tasks.workunit.client.0.smithi079.stdout:inferring bluefs devices from bluestore path 2024-09-01T03:28:08.694 INFO:tasks.workunit.client.0.smithi079.stdout:1 : device size 0xffffe000 : using 0x47078000(1.1 GiB) 2024-09-01T03:28:08.701 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-01T03:28:11.265 INFO:tasks.workunit.client.0.smithi079.stdout:fsck success 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:28:11.306 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:11.307 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-01T03:28:11.308 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:28:11.309 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:28:11.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-01T03:28:11.310 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:28:11.311 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-01T03:28:11.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-01T03:28:11.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-01T03:28:11.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-01T03:28:11.311 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-01T03:28:11.337 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:11.336+0000 7f0d33a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:11.341 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:11.338+0000 7f0d33a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:11.341 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:11.339+0000 7f0d33a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:11.712 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:28:11.713 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:28:11.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:11.714 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:12.138 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:12.424 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:12.422+0000 7f0d33a14640 -1 Falling back to public interface 2024-09-01T03:28:13.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:13.139 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:13.139 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:28:13.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:28:13.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:13.140 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:13.573 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:14.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:14.574 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:14.574 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:28:14.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:28:14.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:14.575 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:15.008 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:16.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:16.009 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:16.009 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:28:16.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:28:16.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:16.010 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:16.444 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:17.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:17.446 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:17.446 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:28:17.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:28:17.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:17.447 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:17.854 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:18.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:18.856 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:18.856 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:28:18.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:28:18.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:18.857 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:19.243 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:20.245 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:20.245 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:28:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:28:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:20.246 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:20.340 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:20.338+0000 7f0d33a14640 -1 osd.0 10 log_to_monitors true 2024-09-01T03:28:20.663 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:21.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:21.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:21.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-01T03:28:21.665 INFO:tasks.workunit.client.0.smithi079.stdout:7 2024-09-01T03:28:21.665 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:21.666 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:22.821 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:23.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:23.822 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:23.823 INFO:tasks.workunit.client.0.smithi079.stdout:8 2024-09-01T03:28:23.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-01T03:28:23.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:23.823 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:24.280 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 13 up_thru 7 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/2256846971,v1:127.0.0.1:6803/2256846971] [v2:127.0.0.1:6804/2256846971,v1:127.0.0.1:6805/2256846971] exists,up 2c6aeb72-e5f9-4471-888d-dcd73903d2a3 2024-09-01T03:28:24.280 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:28:24.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:28:24.281 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:28:24.281 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:28:24.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=143335 2024-09-01T03:28:24.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-01T03:28:24.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-01T03:28:24.282 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-01T03:28:24.283 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-01T03:28:24.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-01T03:28:24.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-01T03:28:24.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-01T03:28:24.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-01T03:28:24.372 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-01T03:28:24.372 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-01T03:28:24.773 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-01T03:28:24.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-01T03:28:24.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-01T03:28:24.774 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-01T03:28:24.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574851 2024-09-01T03:28:24.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574851 2024-09-01T03:28:24.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-55834574851' 2024-09-01T03:28:24.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-01T03:28:24.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-55834574851 2024-09-01T03:28:24.910 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-01T03:28:24.910 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-01T03:28:24.911 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-55834574851 2024-09-01T03:28:24.911 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-01T03:28:24.911 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574851 2024-09-01T03:28:24.912 INFO:tasks.workunit.client.0.smithi079.stdout:waiting osd.0 seq 55834574851 2024-09-01T03:28:24.912 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 55834574851' 2024-09-01T03:28:24.912 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:28:25.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 55834574851 2024-09-01T03:28:25.299 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-01T03:28:26.301 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-01T03:28:26.301 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-01T03:28:26.761 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574852 -lt 55834574851 2024-09-01T03:28:26.762 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-01T03:28:26.762 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:28:26.762 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:28:27.285 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-01T03:28:27.286 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-01T03:28:27.689 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-01T03:28:27.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-01T03:28:27.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-01T03:28:27.690 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:28:28.214 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:28:28.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:28:28.215 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:28:28.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:28:28.320 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:28:28.320 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:28:28.321 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:28:28.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:28:28.322 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:28:28.322 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:28:28.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:28:28.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:28:28.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:28:28.323 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:28:28.324 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:28:28.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:28:28.325 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-01T03:28:28.424 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:28:28.424 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:28.424 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:28.424 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.91822 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:28.426 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:28:28.427 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:28:28.428 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:28:28.428 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:28:28.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:28:28.429 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:28:28.430 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:28:28.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:28:28.430 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:28:28.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:28:28.431 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:28:28.431 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:28:28.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-01T03:28:28.432 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-01T03:28:28.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:28:28.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:28.434 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:28.434 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.91822 2024-09-01T03:28:28.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:28:28.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:28:28.435 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-01T03:28:28.436 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-01T03:28:28.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:28.437 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:28.437 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.91822 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-01T03:28:28.441 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=ede4ade3-9cc8-4639-a776-0b948f751b4b --auth-supported=none ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-01T03:28:28.442 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-01T03:28:28.443 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:28:28.909 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-01T03:28:29.134 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:28:29.136 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:28:29.137 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:29.137 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:29.137 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.91822/ceph-mon.a.asok 2024-09-01T03:28:29.137 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:28:29.137 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.91822/ceph-mon.a.asok config get fsid 2024-09-01T03:28:29.215 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-01T03:28:29.215 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-01T03:28:29.215 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-01T03:28:29.215 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-01T03:28:29.215 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-01T03:28:29.215 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-01T03:28:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-01T03:28:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-01T03:28:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-01T03:28:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:29.216 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.91822/ceph-mon.a.asok 2024-09-01T03:28:29.217 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-01T03:28:29.217 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.91822/ceph-mon.a.asok config get mon_host 2024-09-01T03:28:29.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-01T03:28:29.293 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:29.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-01T03:28:29.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-01T03:28:29.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-01T03:28:29.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-01T03:28:29.294 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-01T03:28:30.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-01T03:28:30.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:28:30.155 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:28:30.155 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:28:30.156 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:30.156 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:30.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:28:30.156 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:28:30.157 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-01T03:28:30.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-01T03:28:30.192 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ede4ade3-9cc8-4639-a776-0b948f751b4b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-01T03:28:30.193 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-01T03:28:30.196 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-01T03:28:30.197 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a5161c50-6269-40e9-8378-d0fa708a60f7 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a5161c50-6269-40e9-8378-d0fa708a60f7' 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stdout:add osd0 a5161c50-6269-40e9-8378-d0fa708a60f7 2024-09-01T03:28:30.198 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-01T03:28:30.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBe39Nm5wwZDRAA8f+1iWRswDpTlhPcLJCG9A== 2024-09-01T03:28:30.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBe39Nm5wwZDRAA8f+1iWRswDpTlhPcLJCG9A=="}' 2024-09-01T03:28:30.221 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a5161c50-6269-40e9-8378-d0fa708a60f7 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-01T03:28:30.484 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:28:30.493 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-01T03:28:30.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ede4ade3-9cc8-4639-a776-0b948f751b4b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBe39Nm5wwZDRAA8f+1iWRswDpTlhPcLJCG9A== --osd-uuid a5161c50-6269-40e9-8378-d0fa708a60f7 2024-09-01T03:28:30.534 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:30.533+0000 7f2f2081d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:30.553 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:30.551+0000 7f2f2081d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:30.554 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:30.552+0000 7f2f2081d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:30.554 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:30.553+0000 7f2f2081d640 -1 bdev(0x556032c52800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-01T03:28:30.554 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:30.553+0000 7f2f2081d640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-01T03:28:38.063 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-01T03:28:38.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-01T03:28:38.064 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-01T03:28:38.065 INFO:tasks.workunit.client.0.smithi079.stdout:adding osd0 key to auth repository 2024-09-01T03:28:38.065 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-01T03:28:38.603 INFO:tasks.workunit.client.0.smithi079.stdout:start osd.0 2024-09-01T03:28:38.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-01T03:28:38.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=33d38768-f0ae-40ae-a77c-ba93a61e6bfa --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=7f5ca465-0279-4eb1-a8ac-b81f1676cfe1 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=ede4ade3-9cc8-4639-a776-0b948f751b4b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.91822/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-01T03:28:38.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-01T03:28:38.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-01T03:28:38.604 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-01T03:28:38.642 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:38.640+0000 7fdd5951b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:38.656 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:38.654+0000 7fdd5951b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:38.657 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:38.656+0000 7fdd5951b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-01T03:28:39.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-01T03:28:39.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-01T03:28:39.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-01T03:28:39.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-01T03:28:39.069 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-01T03:28:39.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:39.070 INFO:tasks.workunit.client.0.smithi079.stdout:0 2024-09-01T03:28:39.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-01T03:28:39.070 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:39.071 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:39.494 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:40.243 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:40.242+0000 7fdd5951b640 -1 Falling back to public interface 2024-09-01T03:28:40.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:40.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:40.496 INFO:tasks.workunit.client.0.smithi079.stdout:1 2024-09-01T03:28:40.496 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-01T03:28:40.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:40.497 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:40.929 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:41.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:41.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:41.930 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-01T03:28:41.931 INFO:tasks.workunit.client.0.smithi079.stdout:2 2024-09-01T03:28:41.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:41.931 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:42.371 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:43.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:43.373 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:43.373 INFO:tasks.workunit.client.0.smithi079.stdout:3 2024-09-01T03:28:43.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-01T03:28:43.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:43.374 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:43.797 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:44.391 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:44.389+0000 7fdd5951b640 -1 osd.0 0 log_to_monitors true 2024-09-01T03:28:44.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:44.798 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:44.798 INFO:tasks.workunit.client.0.smithi079.stdout:4 2024-09-01T03:28:44.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-01T03:28:44.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:44.799 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:46.409 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:47.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:47.410 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:47.411 INFO:tasks.workunit.client.0.smithi079.stdout:5 2024-09-01T03:28:47.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-01T03:28:47.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:47.411 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:48.500 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-01T03:28:48.709 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:28:48.708+0000 7fdd5951b640 -1 bdev(0x555c46aaf800 td/osd-bluefs-volume-ops/0/block) aio_submit retries 8 2024-09-01T03:28:49.501 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-01T03:28:49.501 INFO:tasks.workunit.client.0.smithi079.stdout:6 2024-09-01T03:28:49.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-01T03:28:49.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-01T03:28:49.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-01T03:28:49.502 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-01T03:28:49.922 INFO:tasks.workunit.client.0.smithi079.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2629933227,v1:127.0.0.1:6803/2629933227] [v2:127.0.0.1:6804/2629933227,v1:127.0.0.1:6805/2629933227] exists,up a5161c50-6269-40e9-8378-d0fa708a60f7 2024-09-01T03:28:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-01T03:28:49.922 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-01T03:28:49.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-01T03:28:49.923 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-01T03:28:49.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=146094 2024-09-01T03:28:49.923 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-01T03:28:54.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-01T03:28:54.925 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-01T03:28:55.506 INFO:tasks.workunit.client.0.smithi079.stderr:pool 'foo' already exists 2024-09-01T03:28:55.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-01T03:28:56.519 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-01T03:28:56.565 INFO:tasks.workunit.client.0.smithi079.stdout:hints = 1 2024-09-01T03:28:56.565 INFO:tasks.workunit.client.0.smithi079.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-01T03:28:56.565 INFO:tasks.workunit.client.0.smithi079.stdout:Object prefix: benchmark_data_smithi079_147130 2024-09-01T03:28:56.565 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:28:56.565 INFO:tasks.workunit.client.0.smithi079.stdout: 0 0 0 0 0 0 - 0 2024-09-01T03:28:57.565 INFO:tasks.workunit.client.0.smithi079.stdout: 1 16 160 144 0.562439 0.5625 0.0920839 0.108779 2024-09-01T03:28:58.568 INFO:tasks.workunit.client.0.smithi079.stdout: 2 16 304 288 0.562423 0.5625 0.14265 0.110677 2024-09-01T03:28:59.565 INFO:tasks.workunit.client.0.smithi079.stdout: 3 16 449 433 0.56373 0.566406 0.108512 0.108479 2024-09-01T03:29:00.565 INFO:tasks.workunit.client.0.smithi079.stdout: 4 16 592 576 0.56243 0.558594 0.130549 0.110914 2024-09-01T03:29:01.565 INFO:tasks.workunit.client.0.smithi079.stdout: 5 16 704 688 0.537433 0.4375 0.125629 0.115208 2024-09-01T03:29:02.566 INFO:tasks.workunit.client.0.smithi079.stdout: 6 16 849 833 0.542248 0.566406 0.0837519 0.114249 2024-09-01T03:29:03.566 INFO:tasks.workunit.client.0.smithi079.stdout: 7 16 1008 992 0.553503 0.621094 0.125103 0.112844 2024-09-01T03:29:04.566 INFO:tasks.workunit.client.0.smithi079.stdout: 8 16 1137 1121 0.547296 0.503906 0.0920386 0.113569 2024-09-01T03:29:05.566 INFO:tasks.workunit.client.0.smithi079.stdout: 9 16 1296 1280 0.555486 0.621094 0.125815 0.11215 2024-09-01T03:29:06.566 INFO:tasks.workunit.client.0.smithi079.stdout: 10 16 1425 1409 0.550319 0.503906 0.176104 0.112158 2024-09-01T03:29:07.566 INFO:tasks.workunit.client.0.smithi079.stdout: 11 16 1585 1569 0.557102 0.625 0.0837888 0.111357 2024-09-01T03:29:08.566 INFO:tasks.workunit.client.0.smithi079.stdout: 12 16 1744 1728 0.562429 0.621094 0.0924812 0.110905 2024-09-01T03:29:09.566 INFO:tasks.workunit.client.0.smithi079.stdout: 13 16 1889 1873 0.562729 0.566406 0.0918506 0.110518 2024-09-01T03:29:10.566 INFO:tasks.workunit.client.0.smithi079.stdout: 14 16 2032 2016 0.562428 0.558594 0.13403 0.110938 2024-09-01T03:29:11.567 INFO:tasks.workunit.client.0.smithi079.stdout: 15 16 2177 2161 0.562688 0.566406 0.0922474 0.110603 2024-09-01T03:29:12.567 INFO:tasks.workunit.client.0.smithi079.stdout: 16 16 2305 2289 0.558765 0.5 0.159195 0.1111 2024-09-01T03:29:13.567 INFO:tasks.workunit.client.0.smithi079.stdout: 17 16 2465 2449 0.562655 0.625 0.0837764 0.110609 2024-09-01T03:29:14.568 INFO:tasks.workunit.client.0.smithi079.stdout: 18 16 2608 2592 0.562426 0.558594 0.22645 0.111024 2024-09-01T03:29:15.569 INFO:tasks.workunit.client.0.smithi079.stdout: 19 16 2753 2737 0.562632 0.566406 0.100341 0.110668 2024-09-01T03:29:16.567 INFO:tasks.workunit.client.0.smithi079.stdout:2024-09-01T03:29:16.567265+0000 min lat: 0.0829913 max lat: 0.351276 avg lat: 0.110804 2024-09-01T03:29:16.568 INFO:tasks.workunit.client.0.smithi079.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-01T03:29:16.568 INFO:tasks.workunit.client.0.smithi079.stdout: 20 16 2897 2881 0.562622 0.5625 0.0837894 0.110804 2024-09-01T03:29:17.568 INFO:tasks.workunit.client.0.smithi079.stdout: 21 16 3057 3041 0.565588 0.625 0.108745 0.110049 2024-09-01T03:29:18.568 INFO:tasks.workunit.client.0.smithi079.stdout: 22 16 3185 3169 0.562603 0.5 0.139114 0.110627 2024-09-01T03:29:19.568 INFO:tasks.workunit.client.0.smithi079.stdout: 23 16 3344 3328 0.565142 0.621094 0.0920176 0.110504 2024-09-01T03:29:20.568 INFO:tasks.workunit.client.0.smithi079.stdout: 24 16 3473 3457 0.562587 0.503906 0.083667 0.110897 2024-09-01T03:29:21.568 INFO:tasks.workunit.client.0.smithi079.stdout: 25 16 3633 3617 0.565081 0.625 0.133803 0.110261 2024-09-01T03:29:22.568 INFO:tasks.workunit.client.0.smithi079.stdout: 26 16 3760 3744 0.562426 0.496094 0.117068 0.111064 2024-09-01T03:29:23.569 INFO:tasks.workunit.client.0.smithi079.stdout: 27 16 3920 3904 0.564741 0.625 0.109191 0.110632 2024-09-01T03:29:24.568 INFO:tasks.workunit.client.0.smithi079.stdout: 28 16 4049 4033 0.562566 0.503906 0.109127 0.110773 2024-09-01T03:29:25.569 INFO:tasks.workunit.client.0.smithi079.stdout: 29 16 4208 4192 0.564582 0.621094 0.117413 0.1106 2024-09-01T03:29:26.569 INFO:tasks.workunit.client.0.smithi079.stdout: 30 1 4337 4336 0.564511 0.5625 0.0915174 0.110695 2024-09-01T03:29:27.569 INFO:tasks.workunit.client.0.smithi079.stdout:Total time run: 30.0361 2024-09-01T03:29:27.569 INFO:tasks.workunit.client.0.smithi079.stdout:Total writes made: 4337 2024-09-01T03:29:27.569 INFO:tasks.workunit.client.0.smithi079.stdout:Write size: 4096 2024-09-01T03:29:27.569 INFO:tasks.workunit.client.0.smithi079.stdout:Object size: 4096 2024-09-01T03:29:27.569 INFO:tasks.workunit.client.0.smithi079.stdout:Bandwidth (MB/sec): 0.564035 2024-09-01T03:29:27.569 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Bandwidth: 0.0519514 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Max bandwidth (MB/sec): 0.625 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Min bandwidth (MB/sec): 0.4375 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Average IOPS: 144 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev IOPS: 13.2996 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Max IOPS: 160 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Min IOPS: 112 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Average Latency(s): 0.110691 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Stddev Latency(s): 0.0302893 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Max latency(s): 0.351276 2024-09-01T03:29:27.570 INFO:tasks.workunit.client.0.smithi079.stdout:Min latency(s): 0.0753088 2024-09-01T03:29:27.629 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-01T03:29:32.632 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-01T03:29:32.632 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-01T03:29:32.774 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-01T03:29:32.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-01T03:29:32.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-01T03:29:32.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-01T03:29:32.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-01T03:29:32.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-01T03:29:32.775 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-01T03:29:32.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4277096448 2024-09-01T03:29:32.899 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 146094 2024-09-01T03:29:32.900 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-01T03:29:33.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 146094 2024-09-01T03:29:33.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (146094) - No such process 2024-09-01T03:29:33.901 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-01T03:29:34.312 INFO:tasks.workunit.client.0.smithi079.stderr:osd.0 is already down. 2024-09-01T03:29:34.323 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-01T03:29:34.351 INFO:tasks.workunit.client.0.smithi079.stdout:allocmap bluestore.allocmap 2024-09-01T03:29:36.957 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:29:36.955+0000 7fca666b2980 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-09-01T03:29:36.958 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:29:36.955+0000 7fca666b2980 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-09-01T03:29:36.958 INFO:tasks.workunit.client.0.smithi079.stderr:2024-09-01T03:29:36.955+0000 7fca666b2980 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-09-01T03:29:37.392 INFO:tasks.workunit.client.0.smithi079.stderr:allocmap failed: (1) Operation not permitted 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bluefs-volume-ops 1 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-01T03:29:37.399 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-01T03:29:37.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-01T03:29:37.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-01T03:29:37.506 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-01T03:29:37.506 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-01T03:29:37.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-01T03:29:37.507 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-01T03:29:37.507 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-01T03:29:37.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:29:37.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-01T03:29:37.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-01T03:29:37.509 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-01T03:29:37.509 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:29:37.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-01T03:29:37.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-01T03:29:37.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-01T03:29:37.511 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-01T03:29:37.512 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-01T03:29:37.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-01T03:29:37.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-01T03:29:37.516 INFO:tasks.workunit.client.0.smithi079.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.91822 2024-09-01T03:29:37.516 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.91822 2024-09-01T03:29:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T03:29:37.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-01T03:29:37.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-01T03:29:37.518 INFO:tasks.workunit.client.0.smithi079.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-01T03:29:37.518 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-01T03:29:37.519 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-01T03:29:37.838 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-01T03:29:38.252 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ecdb5351893a4efb9123fe9b87f645a4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5028887063e8eb518c95117479aa913eaa12ac8e/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-01T03:29:38.255 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-01T03:29:38.265 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-01T03:29:38.265 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-01T03:29:38.321 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-01T03:29:38.329 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-01T03:29:38.329 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-01T03:29:38.351 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-09-01T03:29:38.356 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-01T03:29:38.356 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-09-01T03:29:38.356 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 +1255us[+1236us] +/- 61ms 2024-09-01T03:29:38.357 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -114us[ -133us] +/- 54ms 2024-09-01T03:29:38.357 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 -1804us[-1823us] +/- 94ms 2024-09-01T03:29:38.357 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-01T03:29:38.359 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-01T03:29:38.366 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-01T03:29:38.367 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-01T03:29:38.393 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-01T03:29:38.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-01T03:29:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T03:29:38.481 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-01T03:29:38.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-01T03:29:38.509 INFO:teuthology.task.internal:Duration was 2542.417600 seconds 2024-09-01T03:29:38.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-01T03:29:38.543 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-01T03:29:38.543 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-01T03:29:38.582 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-01T03:29:39.086 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-01T03:29:39.086 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-09-01T03:29:39.086 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-01T03:29:39.147 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-01T03:29:39.148 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-01T03:29:39.214 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-01T03:29:39.214 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-01T03:29:39.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-01T03:29:39.325 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-01T03:29:39.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-01T03:29:39.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-01T03:29:39.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:29:39.413 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-09-01T03:29:39.431 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-01T03:29:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T03:29:39.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-01T03:29:39.496 INFO:teuthology.task.internal:Transferring archived files... 2024-09-01T03:29:39.498 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-08-31_23:59:00-rados-wip-bharath2-testing-2024-08-31-2129-distro-default-smithi/7884281/remote/smithi079 2024-09-01T03:29:39.499 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-01T03:29:43.153 INFO:teuthology.task.internal:Removing archive directory... 2024-09-01T03:29:43.154 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-01T03:29:43.212 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-01T03:29:43.222 INFO:teuthology.task.internal:Not uploading archives. 2024-09-01T03:29:43.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-01T03:29:43.268 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-01T03:29:43.269 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-01T03:29:43.285 INFO:teuthology.orchestra.run.smithi079.stdout: 262217 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 03:29 /home/ubuntu/cephtest 2024-09-01T03:29:43.285 INFO:teuthology.orchestra.run.smithi079.stdout: 276888 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 02:51 /home/ubuntu/cephtest/mnt.0 2024-09-01T03:29:43.285 INFO:teuthology.orchestra.run.smithi079.stdout: 276889 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 03:17 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-01T03:29:43.285 INFO:teuthology.orchestra.run.smithi079.stdout: 393217 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 1 03:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-01T03:29:43.286 INFO:teuthology.orchestra.run.smithi079.stdout: 393218 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 1 03:29 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-01T03:29:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-01T03:29:43.287 INFO:teuthology.orchestra.run.smithi079.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-01T03:29:43.287 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-01T03:29:43.292 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-01T03:29:43.301 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-01T03:29:43.335 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi079 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-01T03:29:43.339 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} duration: 2542.4176001548767 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi079 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5028887063e8eb518c95117479aa913eaa12ac8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ecdb5351893a4efb9123fe9b87f645a4 status: fail success: false 2024-09-01T03:29:43.339 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-01T03:29:43.439 INFO:teuthology.run:FAIL