2023-08-16T01:49:35.788 INFO:root:teuthology version: 0.0.1.dev183+g5435569 2023-08-16T01:49:35.789 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369212 --owner scheduled_yuriw@teuthology --name yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi --verbose --description rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} -- /home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369212/orig.config.yaml 2023-08-16T01:49:35.811 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-16T01:49:45.913 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369212 branch: wip-yuri3-testing-2023-08-15-0955 description: rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7369212' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri3-testing-2023-08-15-0955 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b5259484dbc61e8573f27e692560c6c107b9d4c1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b5259484dbc61e8573f27e692560c6c107b9d4c1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri3-testing-2023-08-15-0955 sha1: b5259484dbc61e8573f27e692560c6c107b9d4c1 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7928 sha1: b5259484dbc61e8573f27e692560c6c107b9d4c1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-08-15-0955 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b5259484dbc61e8573f27e692560c6c107b9d4c1 targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54355694caed5e99f46f0c9317b1e9bcf8940415 timestamp: 2023-08-15_18:58:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2224 2023-08-16T01:49:45.913 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa; will attempt to use it 2023-08-16T01:49:45.914 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks 2023-08-16T01:49:45.914 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-16T01:49:45.916 INFO:teuthology.task.internal:Checking packages... 2023-08-16T01:49:45.954 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'b5259484dbc61e8573f27e692560c6c107b9d4c1' 2023-08-16T01:49:45.954 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-16T01:49:45.955 INFO:teuthology.packaging:ref: None 2023-08-16T01:49:45.955 INFO:teuthology.packaging:tag: None 2023-08-16T01:49:45.956 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-08-15-0955 2023-08-16T01:49:45.956 INFO:teuthology.packaging:sha1: b5259484dbc61e8573f27e692560c6c107b9d4c1 2023-08-16T01:49:45.957 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2023-08-15-0955 2023-08-16T01:49:51.187 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5526.gb5259484 2023-08-16T01:49:51.187 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-16T01:49:51.200 INFO:teuthology.task.internal:no buildpackages task found 2023-08-16T01:49:51.201 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-16T01:49:51.217 INFO:teuthology.task.internal:Saving configuration 2023-08-16T01:49:51.226 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-16T01:49:51.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-16T01:49:56.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 01:43:22.341376', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-16T01:49:56.264 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-16T01:49:56.271 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-08-16T01:49:56.272 INFO:teuthology.run_tasks:Running task console_log... 2023-08-16T01:50:01.283 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f043dda8e50>, signals=[15]) 2023-08-16T01:50:01.285 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-16T01:50:01.297 INFO:teuthology.task.internal:Opening connections... 2023-08-16T01:50:01.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2023-08-16T01:50:01.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T01:50:01.373 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-08-16T01:50:01.380 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2023-08-16T01:50:01.446 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-08-16T01:50:01.446 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2023-08-16T01:50:01.505 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Red Hat Enterprise Linux" 2023-08-16T01:50:01.505 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8.6 (Ootpa)" 2023-08-16T01:50:01.505 INFO:teuthology.orchestra.run.smithi150.stdout:ID="rhel" 2023-08-16T01:50:01.505 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="fedora" 2023-08-16T01:50:01.505 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8.6" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.redhat.com/" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-16T01:50:01.506 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-16T01:50:01.507 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T01:50:01.507 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-16T01:50:01.507 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-16T01:50:01.507 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-16T01:50:01.507 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-16T01:50:01.508 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-08-16T01:50:06.535 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-08-16T01:50:06.545 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-08-16T01:50:06.557 INFO:teuthology.task.internal:Checking for old test directory... 2023-08-16T01:50:06.558 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2023-08-16T01:50:06.577 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-08-16T01:50:06.587 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-08-16T01:50:06.587 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2023-08-16T01:50:06.641 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-08-16T01:50:12.211 INFO:teuthology.run_tasks:Running task kernel... 2023-08-16T01:50:12.231 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-08-16T01:50:12.232 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-08-16T01:50:12.233 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-16T01:50:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T01:50:12.253 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2023-08-16T01:50:12.313 INFO:teuthology.orchestra.run.smithi150.stdout:4.18.0-372.9.1.el8.x86_64 2023-08-16T01:50:12.314 INFO:teuthology.task.kernel:Running kernel on smithi150: 4.18.0-372.9.1.el8.x86_64 2023-08-16T01:50:12.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2023-08-16T01:50:13.580 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T01:50:13.580 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-08-16T01:50:14.950 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-08-16T01:50:15.784 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-08-16T01:50:19.349 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 334 kB/s | 24 kB 00:00 2023-08-16T01:50:20.063 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-08-16T01:50:20.108 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T01:50:20.110 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T01:50:20.110 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T01:50:20.224 DEBUG:teuthology.orchestra.run.smithi150:> echo no | sudo yum reinstall kernel || true 2023-08-16T01:50:20.559 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T01:50:20.559 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-08-16T01:50:20.826 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:01 ago on Wed 16 Aug 2023 01:50:19 AM UTC. 2023-08-16T01:50:20.867 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-16T01:50:20.903 INFO:teuthology.orchestra.run.smithi150.stderr:Error: No packages marked for reinstall. 2023-08-16T01:50:20.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum reinstall -y kernel || true 2023-08-16T01:50:21.279 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T01:50:21.280 INFO:teuthology.orchestra.run.smithi150.stdout:Unable to read consumer identity 2023-08-16T01:50:21.536 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:02 ago on Wed 16 Aug 2023 01:50:19 AM UTC. 2023-08-16T01:50:21.575 INFO:teuthology.orchestra.run.smithi150.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-16T01:50:21.617 INFO:teuthology.orchestra.run.smithi150.stderr:Error: No packages marked for reinstall. 2023-08-16T01:50:21.663 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2023-08-16T01:50:21.744 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-08-16T01:50:21.744 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-08-16T01:50:21.744 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-08-16T01:50:21.744 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-08-16T01:50:21.745 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-16T01:50:21.781 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2023-08-16T01:50:21.792 DEBUG:teuthology.parallel:result is None 2023-08-16T01:50:21.793 INFO:teuthology.run_tasks:Running task internal.base... 2023-08-16T01:50:21.803 INFO:teuthology.task.internal:Creating test directory... 2023-08-16T01:50:21.803 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-16T01:50:21.849 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-08-16T01:50:21.856 INFO:teuthology.run_tasks:Running task internal.archive... 2023-08-16T01:50:21.872 INFO:teuthology.task.internal:Creating archive directory... 2023-08-16T01:50:21.872 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-16T01:50:21.909 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-08-16T01:50:21.915 INFO:teuthology.task.internal:Enabling coredump saving... 2023-08-16T01:50:21.916 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-16T01:50:21.977 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T01:50:21.987 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T01:50:21.988 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-08-16T01:50:21.995 INFO:teuthology.task.internal:Configuring sudo... 2023-08-16T01:50:21.996 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-16T01:50:22.055 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-08-16T01:50:22.064 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-08-16T01:50:22.065 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-16T01:50:22.113 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-16T01:50:22.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-16T01:50:22.248 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-16T01:50:27.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-16T01:50:27.368 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T01:50:27.369 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-16T01:50:27.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2023-08-16T01:50:27.538 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-16T01:50:27.640 INFO:teuthology.run_tasks:Running task internal.timer... 2023-08-16T01:50:27.649 INFO:teuthology.task.internal:Starting timer... 2023-08-16T01:50:27.650 INFO:teuthology.run_tasks:Running task pcp... 2023-08-16T01:50:27.664 INFO:teuthology.run_tasks:Running task selinux... 2023-08-16T01:50:27.673 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-08-16T01:50:27.673 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2023-08-16T01:50:27.738 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2023-08-16T01:50:27.740 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-08-16T01:50:27.740 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2023-08-16T01:50:27.762 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2023-08-16T01:50:27.762 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2023-08-16T01:50:27.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-16T01:50:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T01:50:27.831 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-08-16T01:50:27.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2023-08-16T01:50:27.900 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-08-16T01:50:27.910 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-08-16T01:50:27.916 INFO:teuthology.repo_utils:Fetching main from origin 2023-08-16T01:50:33.110 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-08-16T01:50:33.133 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-08-16T01:50:33.134 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-08-16T01:59:26.590 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2023-08-16T01:59:26.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-08-16T01:59:26.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T01:59:26.661 DEBUG:teuthology.orchestra.run.smithi150:> true 2023-08-16T01:59:26.732 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-08-16T01:59:26.733 INFO:teuthology.run_tasks:Running task clock... 2023-08-16T01:59:26.744 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-08-16T01:59:26.745 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-16T01:59:26.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-16T01:59:26.803 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-16T01:59:26.817 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-16T01:59:26.886 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2023-08-16T01:59:26.897 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2023-08-16T01:59:26.912 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-16T01:59:26.926 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-16T01:59:27.003 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-08-16T01:59:27.007 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-16T01:59:27.007 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-08-16T01:59:27.007 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-16T01:59:27.007 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-16T01:59:27.007 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-16T01:59:27.008 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-16T01:59:27.008 INFO:teuthology.run_tasks:Running task install... 2023-08-16T01:59:27.019 DEBUG:teuthology.task.install:project ceph 2023-08-16T01:59:27.020 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b5259484dbc61e8573f27e692560c6c107b9d4c1'}} 2023-08-16T01:59:27.020 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b5259484dbc61e8573f27e692560c6c107b9d4c1'} 2023-08-16T01:59:27.020 INFO:teuthology.task.install:Using flavor: default 2023-08-16T01:59:27.027 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-08-16T01:59:27.027 INFO:teuthology.task.install:extra packages: [] 2023-08-16T01:59:27.027 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b5259484dbc61e8573f27e692560c6c107b9d4c1', 'tag': None, 'wait_for_package': False} 2023-08-16T01:59:27.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b5259484dbc61e8573f27e692560c6c107b9d4c1 2023-08-16T01:59:27.249 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-08-15-0955/b5259484dbc61e8573f27e692560c6c107b9d4c1/centos/8/flavors/default/ 2023-08-16T01:59:27.249 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5526.gb5259484 2023-08-16T01:59:27.355 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-08-15-0955/b5259484dbc61e8573f27e692560c6c107b9d4c1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-08-15-0955/b5259484dbc61e8573f27e692560c6c107b9d4c1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-08-15-0955/b5259484dbc61e8573f27e692560c6c107b9d4c1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-16T01:59:27.355 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T01:59:27.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-16T01:59:27.383 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-08-16T01:59:27.383 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b5259484dbc61e8573f27e692560c6c107b9d4c1/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-16T01:59:27.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-16T01:59:27.529 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-16T01:59:27.601 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2023-08-16T01:59:27.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2023-08-16T01:59:27.972 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T01:59:28.237 INFO:teuthology.orchestra.run.smithi150.stdout:44 files removed 2023-08-16T01:59:28.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2023-08-16T01:59:28.612 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T01:59:29.282 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 267 kB/s | 81 kB 00:00 2023-08-16T01:59:29.675 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-08-16T01:59:30.050 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2023-08-16T01:59:30.210 INFO:teuthology.orchestra.run.smithi150.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-08-16T01:59:31.510 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-08-16T01:59:34.960 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 587 kB/s | 24 kB 00:00 2023-08-16T01:59:35.962 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 MB/s | 53 MB 00:00 2023-08-16T01:59:44.121 INFO:teuthology.orchestra.run.smithi150.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 MB/s | 47 MB 00:00 2023-08-16T01:59:55.326 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T01:59:55.328 INFO:teuthology.orchestra.run.smithi150.stdout:==================================================================================================== 2023-08-16T01:59:55.328 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T01:59:55.328 INFO:teuthology.orchestra.run.smithi150.stdout:==================================================================================================== 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:18.0.0-5526.gb5259484.el8 ceph 14 M 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:18.0.0-5526.gb5259484.el8 ceph 3.7 M 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:18.0.0-5526.gb5259484.el8 ceph 3.6 M 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T01:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:18.0.0-5526.gb5259484.el8 ceph 5.4 M 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:18.0.0-5526.gb5259484.el8 ceph 22 M 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:18.0.0-5526.gb5259484.el8 ceph 24 k 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:18.0.0-5526.gb5259484.el8 ceph 782 k 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-08-16T01:59:55.330 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:18.0.0-5526.gb5259484.el8 ceph 472 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:18.0.0-5526.gb5259484.el8 ceph 6.8 M 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:18.0.0-5526.gb5259484.el8 ceph 45 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:18.0.0-5526.gb5259484.el8 ceph 125 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:18.0.0-5526.gb5259484.el8 ceph 197 k 2023-08-16T01:59:55.331 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:18.0.0-5526.gb5259484.el8 ceph 372 k 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:18.0.0-5526.gb5259484.el8 ceph 366 k 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:18.0.0-5526.gb5259484.el8 ceph 107 k 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T01:59:55.332 INFO:teuthology.orchestra.run.smithi150.stdout:==================================================================================================== 2023-08-16T01:59:55.333 INFO:teuthology.orchestra.run.smithi150.stdout:Install 21 Packages 2023-08-16T01:59:55.333 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2023-08-16T01:59:55.333 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T01:59:55.333 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 61 M 2023-08-16T01:59:55.334 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T01:59:56.664 INFO:teuthology.orchestra.run.smithi150.stdout:(1/23): ceph-radosgw-18.0.0-5526.gb5259484.el8. 12 MB/s | 14 MB 00:01 2023-08-16T01:59:56.756 INFO:teuthology.orchestra.run.smithi150.stdout:(2/23): ceph-base-18.0.0-5526.gb5259484.el8.x86 4.3 MB/s | 5.4 MB 00:01 2023-08-16T01:59:56.789 INFO:teuthology.orchestra.run.smithi150.stdout:(3/23): ceph-selinux-18.0.0-5526.gb5259484.el8. 194 kB/s | 24 kB 00:00 2023-08-16T01:59:56.856 INFO:teuthology.orchestra.run.smithi150.stdout:(4/23): libradosstriper1-18.0.0-5526.gb5259484. 6.9 MB/s | 472 kB 00:00 2023-08-16T01:59:56.923 INFO:teuthology.orchestra.run.smithi150.stdout:(5/23): libcephfs2-18.0.0-5526.gb5259484.el8.x8 4.6 MB/s | 782 kB 00:00 2023-08-16T01:59:57.007 INFO:teuthology.orchestra.run.smithi150.stdout:(6/23): python3-ceph-argparse-18.0.0-5526.gb525 537 kB/s | 45 kB 00:00 2023-08-16T01:59:57.107 INFO:teuthology.orchestra.run.smithi150.stdout:(7/23): python3-ceph-common-18.0.0-5526.gb52594 1.2 MB/s | 125 kB 00:00 2023-08-16T01:59:57.207 INFO:teuthology.orchestra.run.smithi150.stdout:(8/23): python3-cephfs-18.0.0-5526.gb5259484.el 1.9 MB/s | 197 kB 00:00 2023-08-16T01:59:57.299 INFO:teuthology.orchestra.run.smithi150.stdout:(9/23): librgw2-18.0.0-5526.gb5259484.el8.x86_6 15 MB/s | 6.8 MB 00:00 2023-08-16T01:59:57.349 INFO:teuthology.orchestra.run.smithi150.stdout:(10/23): python3-rados-18.0.0-5526.gb5259484.el 2.6 MB/s | 372 kB 00:00 2023-08-16T01:59:57.375 INFO:teuthology.orchestra.run.smithi150.stdout:(11/23): python3-rbd-18.0.0-5526.gb5259484.el8. 4.8 MB/s | 366 kB 00:00 2023-08-16T01:59:57.417 INFO:teuthology.orchestra.run.smithi150.stdout:(12/23): python3-rgw-18.0.0-5526.gb5259484.el8. 1.6 MB/s | 107 kB 00:00 2023-08-16T01:59:57.525 INFO:teuthology.orchestra.run.smithi150.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 543 kB/s | 59 kB 00:00 2023-08-16T01:59:57.551 INFO:teuthology.orchestra.run.smithi150.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2023-08-16T01:59:57.593 INFO:teuthology.orchestra.run.smithi150.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-08-16T01:59:57.693 INFO:teuthology.orchestra.run.smithi150.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2023-08-16T01:59:57.801 INFO:teuthology.orchestra.run.smithi150.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2023-08-16T01:59:58.069 INFO:teuthology.orchestra.run.smithi150.stdout:(18/23): ceph-common-18.0.0-5526.gb5259484.el8. 8.6 MB/s | 22 MB 00:02 2023-08-16T01:59:58.094 INFO:teuthology.orchestra.run.smithi150.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 93 kB/s | 47 kB 00:00 2023-08-16T01:59:58.119 INFO:teuthology.orchestra.run.smithi150.stdout:(20/23): python3-prettytable-0.7.2-14.el8.noarc 139 kB/s | 44 kB 00:00 2023-08-16T01:59:58.202 INFO:teuthology.orchestra.run.smithi150.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2023-08-16T01:59:58.378 INFO:teuthology.orchestra.run.smithi150.stdout:(22/23): librbd1-18.0.0-5526.gb5259484.el8.x86_ 14 MB/s | 3.6 MB 00:00 2023-08-16T01:59:58.653 INFO:teuthology.orchestra.run.smithi150.stdout:(23/23): librados2-18.0.0-5526.gb5259484.el8.x8 7.0 MB/s | 3.7 MB 00:00 2023-08-16T01:59:58.655 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T01:59:58.656 INFO:teuthology.orchestra.run.smithi150.stdout:Total 18 MB/s | 61 MB 00:03 2023-08-16T01:59:58.703 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T01:59:58.749 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T01:59:58.750 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T01:59:59.087 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T01:59:59.091 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:00:00.317 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:00:00.582 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-08-16T02:00:00.889 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-08-16T02:00:01.289 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-08-16T02:00:01.514 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:18.0.0-5526.gb5259484.el8.x86_64 3/25 2023-08-16T02:00:01.612 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:18.0.0-5526.gb5259484.el8.x86_64 3/25 2023-08-16T02:00:01.804 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:18.0.0-5526.gb5259484.el8.x86_64 4/25 2023-08-16T02:00:02.071 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-08-16T02:00:02.272 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:18.0.0-5526.gb5259484.el8.x86_64 6/25 2023-08-16T02:00:02.665 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:18.0.0-5526.gb5259484.el8.x86_64 6/25 2023-08-16T02:00:02.855 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:18.0.0-5526.gb5259484.el8.x86_64 7/25 2023-08-16T02:00:02.954 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:18.0.0-5526.gb5259484.el8.x86_64 7/25 2023-08-16T02:00:03.139 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-08-16T02:00:03.203 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-08-16T02:00:03.379 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-08-16T02:00:03.533 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:18.0.0-5526.gb5259484.el8. 10/25 2023-08-16T02:00:03.749 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:18.0.0-5526.gb5259484.el8.x86_64 11/25 2023-08-16T02:00:03.948 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:18.0.0-5526.gb5259484.el8.x86_64 12/25 2023-08-16T02:00:04.105 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:18.0.0-5526.gb5259484.el8.x86_6 13/25 2023-08-16T02:00:04.176 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5526.gb5259484.el8.x86_6 13/25 2023-08-16T02:00:04.347 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-08-16T02:00:04.539 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-08-16T02:00:05.353 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-08-16T02:00:05.563 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:18.0.0-5526.gb5259484.el8.x86_64 17/25 2023-08-16T02:00:05.648 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:18.0.0-5526.gb5259484.el8.x86_64 17/25 2023-08-16T02:00:06.133 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:18.0.0-5526.gb5259484.el8.x86_64 18/25 2023-08-16T02:00:06.339 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:18.0.0-5526.gb5259484.el8.x8 19/25 2023-08-16T02:00:09.986 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-5526.gb5259484.el8.x86_64 20/25 2023-08-16T02:00:10.263 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:18.0.0-5526.gb5259484.el8.x86_64 20/25 2023-08-16T02:00:10.860 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:18.0.0-5526.gb5259484.el8.x86_64 20/25 2023-08-16T02:00:11.112 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:18.0.0-5526.gb5259484.el8.x86_64 21/25 2023-08-16T02:00:11.211 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:18.0.0-5526.gb5259484.el8.x86_64 21/25 2023-08-16T02:00:11.413 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:18.0.0-5526.gb5259484.el8.x86_64 22/25 2023-08-16T02:00:30.208 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5526.gb5259484.el8.x86_64 22/25 2023-08-16T02:00:30.208 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2023-08-16T02:00:30.208 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /run 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2023-08-16T02:00:30.209 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /dev 2023-08-16T02:00:30.210 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:31.642 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:18.0.0-5526.gb5259484.el8.x86_64 23/25 2023-08-16T02:00:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5526.gb5259484.el8.x86_64 23/25 2023-08-16T02:00:31.664 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-08-16T02:00:31.862 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-08-16T02:00:31.862 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-08-16T02:00:32.487 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-08-16T02:00:32.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:18.0.0-5526.gb5259484.el8.x86_64 1/25 2023-08-16T02:00:32.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:18.0.0-5526.gb5259484.el8.x86_64 2/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:18.0.0-5526.gb5259484.el8.x86_64 3/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:18.0.0-5526.gb5259484.el8.x86_64 4/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:18.0.0-5526.gb5259484.el8.x86_64 5/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:18.0.0-5526.gb5259484.el8.x86_6 6/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:18.0.0-5526.gb5259484.el8.x86_64 7/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5526.gb5259484.el8. 8/25 2023-08-16T02:00:32.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:18.0.0-5526.gb5259484.el8.x8 9/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:18.0.0-5526.gb5259484.el8.x86_64 10/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:18.0.0-5526.gb5259484.el8.x86_64 11/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:18.0.0-5526.gb5259484.el8.x86_64 12/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:18.0.0-5526.gb5259484.el8.x86_64 13/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-08-16T02:00:32.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:18.0.0-5526.gb5259484.el8.x86_64 22/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-08-16T02:00:32.490 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:18.0.0-5526.gb5259484.el8.x86_64 24/25 2023-08-16T02:00:33.861 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-08-16T02:00:33.861 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:50 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:00:38.309 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:00:38.503 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.504 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.2-3.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.505 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.3.1-3.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-08-16T02:00:38.506 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.507 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.507 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:38.507 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.13.0-2.el8.x86_64 2023-08-16T02:00:38.507 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:38.507 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:00:38.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2023-08-16T02:00:39.032 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:00:39.956 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:56 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:00:41.413 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:00:41.413 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2023-08-16T02:00:41.414 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:00:41.414 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2023-08-16T02:00:41.414 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:00:41.414 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:18.0.0-5526.gb5259484.el8 ceph 48 M 2023-08-16T02:00:41.414 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:00:41.414 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:18.0.0-5526.gb5259484.el8 ceph 139 k 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2023-08-16T02:00:41.415 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-08-16T02:00:41.416 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:41.416 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 49 M 2023-08-16T02:00:41.416 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 218 M 2023-08-16T02:00:41.417 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:00:41.459 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2023-08-16T02:00:41.635 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): libcephsqlite-18.0.0-5526.gb5259484.el8. 635 kB/s | 139 kB 00:00 2023-08-16T02:00:41.718 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2023-08-16T02:00:41.744 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-08-16T02:00:41.919 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): jq-1.6-3.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-08-16T02:00:43.936 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): ceph-test-18.0.0-5526.gb5259484.el8.x86_ 19 MB/s | 48 MB 00:02 2023-08-16T02:00:43.938 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:00:43.938 INFO:teuthology.orchestra.run.smithi150.stdout:Total 19 MB/s | 49 MB 00:02 2023-08-16T02:00:43.976 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:00:43.997 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:00:43.998 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:00:44.263 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:00:44.266 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:00:44.739 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:00:44.908 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-08-16T02:00:44.973 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-08-16T02:00:45.178 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-08-16T02:00:45.325 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-08-16T02:00:45.529 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-08-16T02:00:45.724 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:18.0.0-5526.gb5259484.el8.x86_64 5/6 2023-08-16T02:00:50.357 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5526.gb5259484.el8.x86_64 5/6 2023-08-16T02:00:50.571 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:18.0.0-5526.gb5259484.el8.x86_64 6/6 2023-08-16T02:00:50.815 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:18.0.0-5526.gb5259484.el8.x86_64 6/6 2023-08-16T02:00:50.816 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:18.0.0-5526.gb5259484.el8.x86_64 1/6 2023-08-16T02:00:50.816 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:18.0.0-5526.gb5259484.el8.x86_64 2/6 2023-08-16T02:00:50.816 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-08-16T02:00:50.816 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-08-16T02:00:50.816 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-08-16T02:00:51.872 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-08-16T02:00:51.872 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:08 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:00:54.512 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:00:54.693 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:54.693 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:00:54.694 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:54.694 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-3.el8.x86_64 2023-08-16T02:00:54.694 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:00:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-08-16T02:00:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-1.el8.x86_64 2023-08-16T02:00:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-08-16T02:00:54.696 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:54.696 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:00:54.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2023-08-16T02:00:55.166 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:00:56.064 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:13 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:00:57.522 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:00:57.526 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================= 2023-08-16T02:00:57.526 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:00:57.526 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================= 2023-08-16T02:00:57.527 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:00:57.527 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:18.0.0-5526.gb5259484.el8 ceph 6.4 k 2023-08-16T02:00:57.527 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:00:57.528 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:18.0.0-5526.gb5259484.el8 ceph 2.3 M 2023-08-16T02:00:57.528 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:18.0.0-5526.gb5259484.el8 ceph 1.6 M 2023-08-16T02:00:57.528 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 241 k 2023-08-16T02:00:57.528 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:18.0.0-5526.gb5259484.el8 ceph 4.6 M 2023-08-16T02:00:57.529 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:18.0.0-5526.gb5259484.el8 ceph 18 M 2023-08-16T02:00:57.529 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-08-16T02:00:57.530 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-08-16T02:00:57.530 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-08-16T02:00:57.530 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-08-16T02:00:57.531 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-08-16T02:00:57.531 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-08-16T02:00:57.531 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-08-16T02:00:57.531 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-08-16T02:00:57.532 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-08-16T02:00:57.532 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-08-16T02:00:57.532 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-08-16T02:00:57.533 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-08-16T02:00:57.533 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-08-16T02:00:57.533 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-08-16T02:00:57.534 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-08-16T02:00:57.534 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-08-16T02:00:57.534 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-08-16T02:00:57.534 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-08-16T02:00:57.535 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-08-16T02:00:57.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-08-16T02:00:57.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-08-16T02:00:57.536 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-08-16T02:00:57.537 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-08-16T02:00:57.537 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-08-16T02:00:57.537 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-08-16T02:00:57.538 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-08-16T02:00:57.538 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:57.538 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:00:57.539 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================= 2023-08-16T02:00:57.539 INFO:teuthology.orchestra.run.smithi150.stdout:Install 32 Packages 2023-08-16T02:00:57.539 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:00:57.540 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 30 M 2023-08-16T02:00:57.541 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 120 M 2023-08-16T02:00:57.541 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:00:58.029 INFO:teuthology.orchestra.run.smithi150.stdout:(1/32): ceph-18.0.0-5526.gb5259484.el8.x86_64.r 53 kB/s | 6.4 kB 00:00 2023-08-16T02:00:58.422 INFO:teuthology.orchestra.run.smithi150.stdout:(2/32): ceph-mon-18.0.0-5526.gb5259484.el8.x86_ 12 MB/s | 4.6 MB 00:00 2023-08-16T02:00:58.480 INFO:teuthology.orchestra.run.smithi150.stdout:(3/32): ceph-mgr-18.0.0-5526.gb5259484.el8.x86_ 2.7 MB/s | 1.6 MB 00:00 2023-08-16T02:00:58.597 INFO:teuthology.orchestra.run.smithi150.stdout:(4/32): ceph-mgr-modules-core-18.0.0-5526.gb525 2.0 MB/s | 241 kB 00:00 2023-08-16T02:00:58.712 INFO:teuthology.orchestra.run.smithi150.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 380 kB/s | 44 kB 00:00 2023-08-16T02:00:58.772 INFO:teuthology.orchestra.run.smithi150.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 3.0 MB/s | 185 kB 00:00 2023-08-16T02:00:58.831 INFO:teuthology.orchestra.run.smithi150.stdout:(7/32): ceph-mds-18.0.0-5526.gb5259484.el8.x86_ 2.5 MB/s | 2.3 MB 00:00 2023-08-16T02:00:58.856 INFO:teuthology.orchestra.run.smithi150.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2023-08-16T02:00:58.898 INFO:teuthology.orchestra.run.smithi150.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 299 kB/s | 11 kB 00:00 2023-08-16T02:00:58.941 INFO:teuthology.orchestra.run.smithi150.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 427 kB/s | 18 kB 00:00 2023-08-16T02:00:58.974 INFO:teuthology.orchestra.run.smithi150.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2023-08-16T02:00:58.999 INFO:teuthology.orchestra.run.smithi150.stdout:(12/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2023-08-16T02:00:59.024 INFO:teuthology.orchestra.run.smithi150.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-08-16T02:00:59.050 INFO:teuthology.orchestra.run.smithi150.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-08-16T02:00:59.075 INFO:teuthology.orchestra.run.smithi150.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-08-16T02:00:59.100 INFO:teuthology.orchestra.run.smithi150.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-08-16T02:00:59.126 INFO:teuthology.orchestra.run.smithi150.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2023-08-16T02:00:59.151 INFO:teuthology.orchestra.run.smithi150.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 584 kB/s | 29 kB 00:00 2023-08-16T02:00:59.177 INFO:teuthology.orchestra.run.smithi150.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 550 kB/s | 27 kB 00:00 2023-08-16T02:00:59.210 INFO:teuthology.orchestra.run.smithi150.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 4.3 MB/s | 255 kB 00:00 2023-08-16T02:00:59.235 INFO:teuthology.orchestra.run.smithi150.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2023-08-16T02:00:59.260 INFO:teuthology.orchestra.run.smithi150.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-08-16T02:00:59.311 INFO:teuthology.orchestra.run.smithi150.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-08-16T02:00:59.495 INFO:teuthology.orchestra.run.smithi150.stdout:(24/32): ceph-osd-18.0.0-5526.gb5259484.el8.x86 17 MB/s | 18 MB 00:01 2023-08-16T02:00:59.678 INFO:teuthology.orchestra.run.smithi150.stdout:(25/32): python3-cryptography-3.2.1-5.el8.x86_6 1.5 MB/s | 559 kB 00:00 2023-08-16T02:00:59.703 INFO:teuthology.orchestra.run.smithi150.stdout:(26/32): python3-pycparser-2.14-14.el8.noarch.r 246 kB/s | 109 kB 00:00 2023-08-16T02:00:59.729 INFO:teuthology.orchestra.run.smithi150.stdout:(27/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.0 MB/s | 238 kB 00:00 2023-08-16T02:00:59.745 INFO:teuthology.orchestra.run.smithi150.stdout:(28/32): python3-pytz-2017.2-9.el8.noarch.rpm 808 kB/s | 54 kB 00:00 2023-08-16T02:01:00.221 INFO:teuthology.orchestra.run.smithi150.stdout:(29/32): python3-mako-1.0.6-13.el8.noarch.rpm 319 kB/s | 157 kB 00:00 2023-08-16T02:01:00.339 INFO:teuthology.orchestra.run.smithi150.stdout:(30/32): python3-markupsafe-0.23-19.el8.x86_64. 61 kB/s | 39 kB 00:00 2023-08-16T02:01:00.472 INFO:teuthology.orchestra.run.smithi150.stdout:(31/32): python3-werkzeug-0.12.2-4.el8.noarch.r 630 kB/s | 457 kB 00:00 2023-08-16T02:01:00.630 INFO:teuthology.orchestra.run.smithi150.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 253 kB/s | 103 kB 00:00 2023-08-16T02:01:00.634 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:01:00.634 INFO:teuthology.orchestra.run.smithi150.stdout:Total 9.7 MB/s | 30 MB 00:03 2023-08-16T02:01:00.725 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:01:00.762 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:01:00.762 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:01:00.975 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:01:00.976 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:01:02.141 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:01:02.394 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-08-16T02:01:02.697 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-08-16T02:01:02.918 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-08-16T02:01:03.133 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-08-16T02:01:03.335 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-08-16T02:01:03.567 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-08-16T02:01:03.791 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-08-16T02:01:04.178 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-08-16T02:01:04.450 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-08-16T02:01:04.638 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-08-16T02:01:04.838 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-08-16T02:01:05.030 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-08-16T02:01:05.258 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-08-16T02:01:05.429 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-08-16T02:01:05.593 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-08-16T02:01:05.783 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-08-16T02:01:05.950 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-08-16T02:01:06.103 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-08-16T02:01:06.592 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-08-16T02:01:06.809 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-08-16T02:01:07.036 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-08-16T02:01:07.291 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-08-16T02:01:07.543 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-08-16T02:01:07.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-08-16T02:01:08.049 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-08-16T02:01:08.351 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-08-16T02:01:08.675 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5526.gb5259484.el8. 27/32 2023-08-16T02:01:08.821 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:18.0.0-5526.gb5259484.el8.x86_64 28/32 2023-08-16T02:01:10.408 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5526.gb5259484.el8.x86_64 28/32 2023-08-16T02:01:10.620 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:18.0.0-5526.gb5259484.el8.x86_64 29/32 2023-08-16T02:01:11.077 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:18.0.0-5526.gb5259484.el8.x86_64 29/32 2023-08-16T02:01:11.246 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:18.0.0-5526.gb5259484.el8.x86_64 30/32 2023-08-16T02:01:11.461 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:18.0.0-5526.gb5259484.el8.x86_64 30/32 2023-08-16T02:01:11.655 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:18.0.0-5526.gb5259484.el8.x86_64 31/32 2023-08-16T02:01:11.673 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:18.0.0-5526.gb5259484.el8.x86_64 31/32 2023-08-16T02:01:11.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:18.0.0-5526.gb5259484.el8.x86_64 32/32 2023-08-16T02:01:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:18.0.0-5526.gb5259484.el8.x86_64 32/32 2023-08-16T02:01:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:18.0.0-5526.gb5259484.el8.x86_64 1/32 2023-08-16T02:01:12.132 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:18.0.0-5526.gb5259484.el8.x86_64 2/32 2023-08-16T02:01:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:18.0.0-5526.gb5259484.el8.x86_64 3/32 2023-08-16T02:01:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:18.0.0-5526.gb5259484.el8.x86_64 4/32 2023-08-16T02:01:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:18.0.0-5526.gb5259484.el8.x86_64 5/32 2023-08-16T02:01:12.133 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5526.gb5259484.el8. 6/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-08-16T02:01:12.134 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-08-16T02:01:12.135 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-08-16T02:01:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-08-16T02:01:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-08-16T02:01:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-08-16T02:01:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-08-16T02:01:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-08-16T02:01:12.136 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-08-16T02:01:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-08-16T02:01:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-08-16T02:01:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-08-16T02:01:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-08-16T02:01:12.137 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-08-16T02:01:13.620 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-08-16T02:01:13.620 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:30 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:16.606 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:01:16.794 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:16.795 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:01:16.795 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:01:16.795 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:01:16.795 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:01:16.795 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-08-16T02:01:16.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-6.2-6.el8.noarch 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.0.6-13.el8.noarch 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-08-16T02:01:16.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-2.6-1.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.14-14.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2017.2-9.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-08-16T02:01:16.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-08-16T02:01:16.799 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:16.800 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:01:16.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2023-08-16T02:01:17.285 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:01:18.280 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:35 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:19.575 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:01:19.687 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:01:19.688 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:01:19.688 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:01:19.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2023-08-16T02:01:20.151 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:01:20.987 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:37 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:22.449 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:01:22.450 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:01:22.450 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:01:22.450 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:01:22.450 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:01:22.450 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 170 k 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 170 k 2023-08-16T02:01:22.451 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 687 k 2023-08-16T02:01:22.452 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:01:23.196 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-18.0.0-5526.gb5259484.el8.noarch.rpm 228 kB/s | 170 kB 00:00 2023-08-16T02:01:23.197 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:01:23.197 INFO:teuthology.orchestra.run.smithi150.stdout:Total 228 kB/s | 170 kB 00:00 2023-08-16T02:01:23.198 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:01:23.207 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:01:23.207 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:01:23.209 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:01:23.214 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:01:23.678 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:01:25.362 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:01:25.498 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:01:25.665 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:01:26.729 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:01:26.730 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:43 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:29.231 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:01:29.409 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:29.410 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:01:29.410 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:01:29.410 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:29.410 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:01:29.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2023-08-16T02:01:29.908 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:01:30.782 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:47 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:32.161 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:01:32.162 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:01:32.162 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-08-16T02:01:32.162 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:01:32.162 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:01:32.162 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5526.gb5259484.el8 ceph 167 k 2023-08-16T02:01:32.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:32.163 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:01:32.163 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:01:32.163 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:01:32.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:32.164 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 167 k 2023-08-16T02:01:32.164 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 512 k 2023-08-16T02:01:32.164 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:01:32.477 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-18.0.0-5526.gb52594 531 kB/s | 167 kB 00:00 2023-08-16T02:01:32.478 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:01:32.478 INFO:teuthology.orchestra.run.smithi150.stdout:Total 529 kB/s | 167 kB 00:00 2023-08-16T02:01:32.478 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:01:32.489 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:01:32.490 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:01:32.541 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:01:32.545 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:01:32.660 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:01:32.811 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5526.gb5259484. 1/1 2023-08-16T02:01:33.163 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5526.gb5259484. 1/1 2023-08-16T02:01:34.230 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5526.gb5259484. 1/1 2023-08-16T02:01:34.231 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:51 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:37.152 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:01:37.335 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:37.335 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:01:37.335 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:01:37.335 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:37.336 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:01:37.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2023-08-16T02:01:37.825 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:01:38.647 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:55 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:39.925 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:01:40.035 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:01:40.035 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:01:40.036 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:01:40.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2023-08-16T02:01:40.516 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:01:41.958 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:58 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:43.361 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:01:43.362 INFO:teuthology.orchestra.run.smithi150.stdout:=================================================================================================== 2023-08-16T02:01:43.362 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:01:43.362 INFO:teuthology.orchestra.run.smithi150.stdout:=================================================================================================== 2023-08-16T02:01:43.363 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:01:43.363 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 1.7 M 2023-08-16T02:01:43.363 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:01:43.363 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 23 k 2023-08-16T02:01:43.364 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 15 k 2023-08-16T02:01:43.364 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-16T02:01:43.364 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-08-16T02:01:43.365 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-08-16T02:01:43.365 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:43.365 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:01:43.365 INFO:teuthology.orchestra.run.smithi150.stdout:=================================================================================================== 2023-08-16T02:01:43.366 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2023-08-16T02:01:43.366 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:43.367 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.0 M 2023-08-16T02:01:43.368 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 8.4 M 2023-08-16T02:01:43.368 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:01:43.575 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): ceph-grafana-dashboards-18.0.0-5526.gb52 160 kB/s | 23 kB 00:00 2023-08-16T02:01:43.674 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): ceph-prometheus-alerts-18.0.0-5526.gb525 59 kB/s | 15 kB 00:00 2023-08-16T02:01:43.776 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2023-08-16T02:01:43.876 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): ceph-mgr-dashboard-18.0.0-5526.gb5259484 3.9 MB/s | 1.7 MB 00:00 2023-08-16T02:01:43.909 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 320 kB/s | 43 kB 00:00 2023-08-16T02:01:43.975 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 654 kB/s | 196 kB 00:00 2023-08-16T02:01:43.978 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:01:43.978 INFO:teuthology.orchestra.run.smithi150.stdout:Total 3.3 MB/s | 2.0 MB 00:00 2023-08-16T02:01:43.989 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:01:43.998 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:01:43.998 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:01:44.052 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:01:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:01:44.308 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:01:44.493 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-08-16T02:01:44.689 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-08-16T02:01:44.861 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-08-16T02:01:44.998 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5526.gb5259484.el8.n 4/6 2023-08-16T02:01:45.697 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5526.gb5259484.el8. 5/6 2023-08-16T02:01:45.963 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5526.gb5259484.el8.noarc 6/6 2023-08-16T02:01:46.181 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5526.gb5259484.el8.noarc 6/6 2023-08-16T02:01:46.181 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5526.gb5259484.el8. 1/6 2023-08-16T02:01:46.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5526.gb5259484.el8.noarc 2/6 2023-08-16T02:01:46.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5526.gb5259484.el8.n 3/6 2023-08-16T02:01:46.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-08-16T02:01:46.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-08-16T02:01:47.228 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-08-16T02:01:47.229 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:04 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:49.767 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:01:49.945 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:49.945 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:01:49.945 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:01:49.945 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:01:49.945 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:01:49.945 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-08-16T02:01:49.946 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-08-16T02:01:49.946 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.4.1-12.el8.noarch 2023-08-16T02:01:49.946 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:49.946 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:01:50.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2023-08-16T02:01:50.444 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:01:51.893 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:08 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:01:53.302 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:01:53.303 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================================= 2023-08-16T02:01:53.303 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:01:53.304 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================================= 2023-08-16T02:01:53.304 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:01:53.304 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 7.4 M 2023-08-16T02:01:53.305 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:01:53.305 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-08-16T02:01:53.305 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-08-16T02:01:53.305 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-08-16T02:01:53.305 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-08-16T02:01:53.305 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================================= 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout:Install 8 Packages 2023-08-16T02:01:53.306 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:01:53.307 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 35 M 2023-08-16T02:01:53.307 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 210 M 2023-08-16T02:01:53.307 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:01:53.399 INFO:teuthology.orchestra.run.smithi150.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 1.8 MB/s | 171 kB 00:00 2023-08-16T02:01:53.440 INFO:teuthology.orchestra.run.smithi150.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 4.7 MB/s | 644 kB 00:00 2023-08-16T02:01:53.632 INFO:teuthology.orchestra.run.smithi150.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-08-16T02:01:53.791 INFO:teuthology.orchestra.run.smithi150.stdout:(4/8): python3-scipy-1.0.0-21.module+el8.5.0+10 35 MB/s | 14 MB 00:00 2023-08-16T02:01:53.841 INFO:teuthology.orchestra.run.smithi150.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-08-16T02:01:53.958 INFO:teuthology.orchestra.run.smithi150.stdout:(6/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 29 MB/s | 4.9 MB 00:00 2023-08-16T02:01:54.050 INFO:teuthology.orchestra.run.smithi150.stdout:(7/8): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-08-16T02:01:54.333 INFO:teuthology.orchestra.run.smithi150.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-552 7.2 MB/s | 7.4 MB 00:01 2023-08-16T02:01:54.334 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:01:54.335 INFO:teuthology.orchestra.run.smithi150.stdout:Total 34 MB/s | 35 MB 00:01 2023-08-16T02:01:54.573 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:01:54.592 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:01:54.593 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:01:54.885 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:01:54.889 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:01:55.451 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:01:55.623 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-08-16T02:01:55.742 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-08-16T02:01:56.300 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-16T02:01:56.918 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-16T02:01:57.082 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-08-16T02:01:57.682 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-08-16T02:01:57.865 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-08-16T02:01:58.552 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-08-16T02:01:58.836 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-08-16T02:01:58.990 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-16T02:02:00.755 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-16T02:02:02.013 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-08-16T02:02:02.174 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5526.gb525948 8/8 2023-08-16T02:02:02.437 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5526.gb525948 8/8 2023-08-16T02:02:02.437 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5526.gb525948 1/8 2023-08-16T02:02:02.438 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-16T02:02:02.438 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-08-16T02:02:02.438 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-08-16T02:02:02.438 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-08-16T02:02:02.438 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-16T02:02:02.438 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-08-16T02:02:03.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-08-16T02:02:03.489 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:20 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:05.967 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:02:06.149 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:06.149 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:02:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:02:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-08-16T02:02:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-08-16T02:02:06.150 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.15-3.el8.x86_64 2023-08-16T02:02:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-08-16T02:02:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-08-16T02:02:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-08-16T02:02:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-08-16T02:02:06.151 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:06.152 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:06.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2023-08-16T02:02:06.630 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:07.492 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:24 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:08.878 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:08.880 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================== 2023-08-16T02:02:08.880 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:02:08.880 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================== 2023-08-16T02:02:08.880 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 47 k 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-08-16T02:02:08.881 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-08-16T02:02:08.882 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:08.883 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:02:08.883 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================================== 2023-08-16T02:02:08.883 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2023-08-16T02:02:08.883 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:08.883 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 1.9 M 2023-08-16T02:02:08.884 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 25 M 2023-08-16T02:02:08.884 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:02:09.439 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 75 kB/s | 16 kB 00:00 2023-08-16T02:02:09.502 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 118 kB/s | 33 kB 00:00 2023-08-16T02:02:09.531 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): ceph-mgr-rook-18.0.0-5526.gb5259484.el8 154 kB/s | 47 kB 00:00 2023-08-16T02:02:09.573 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 607 kB/s | 81 kB 00:00 2023-08-16T02:02:09.699 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 489 kB/s | 61 kB 00:00 2023-08-16T02:02:09.757 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 735 kB/s | 43 kB 00:00 2023-08-16T02:02:09.791 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): python3-rsa-4.9-2.el8.noarch.rpm 248 kB/s | 64 kB 00:00 2023-08-16T02:02:09.816 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.6 MB/s | 155 kB 00:00 2023-08-16T02:02:09.850 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 3.4 MB/s | 1.2 MB 00:00 2023-08-16T02:02:09.875 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 500 kB/s | 27 kB 00:00 2023-08-16T02:02:09.909 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 938 kB/s | 110 kB 00:00 2023-08-16T02:02:09.934 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 377 kB/s | 20 kB 00:00 2023-08-16T02:02:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2023-08-16T02:02:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:02:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.8 MB/s | 1.9 MB 00:01 2023-08-16T02:02:10.012 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:02:10.028 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:02:10.028 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:02:10.098 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:02:10.103 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:02:10.581 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:02:10.830 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-08-16T02:02:11.066 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-08-16T02:02:11.262 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-08-16T02:02:11.412 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-08-16T02:02:11.618 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-08-16T02:02:11.893 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-08-16T02:02:12.099 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-08-16T02:02:12.279 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-08-16T02:02:12.444 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-08-16T02:02:12.642 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-08-16T02:02:13.712 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-08-16T02:02:13.922 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-08-16T02:02:14.071 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:18.0.0-5526.gb5259484.el8.noarch 13/13 2023-08-16T02:02:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5526.gb5259484.el8.noarch 13/13 2023-08-16T02:02:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5526.gb5259484.el8.noarch 1/13 2023-08-16T02:02:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-08-16T02:02:14.315 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-08-16T02:02:14.316 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-08-16T02:02:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-08-16T02:02:14.317 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-08-16T02:02:15.452 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-08-16T02:02:15.452 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:32 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:17.838 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:02:18.021 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:18.021 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:02:18.021 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-08-16T02:02:18.022 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el8.noarch 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:18.023 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:18.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2023-08-16T02:02:18.506 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:19.297 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:36 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:20.690 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:20.692 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================================== 2023-08-16T02:02:20.692 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:02:20.692 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================================== 2023-08-16T02:02:20.692 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:02:20.693 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 131 k 2023-08-16T02:02:20.693 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:02:20.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-08-16T02:02:20.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-08-16T02:02:20.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-08-16T02:02:20.693 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================================== 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout:Install 5 Packages 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 6.0 M 2023-08-16T02:02:20.694 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2023-08-16T02:02:20.695 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:02:21.125 INFO:teuthology.orchestra.run.smithi150.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 588 kB/s | 60 kB 00:00 2023-08-16T02:02:21.183 INFO:teuthology.orchestra.run.smithi150.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.9 MB/s | 476 kB 00:00 2023-08-16T02:02:21.217 INFO:teuthology.orchestra.run.smithi150.stdout:(3/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.7 MB/s | 538 kB 00:00 2023-08-16T02:02:21.359 INFO:teuthology.orchestra.run.smithi150.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2023-08-16T02:02:21.492 INFO:teuthology.orchestra.run.smithi150.stdout:(5/5): ceph-mgr-cephadm-18.0.0-5526.gb5259484.e 278 kB/s | 131 kB 00:00 2023-08-16T02:02:21.493 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:02:21.493 INFO:teuthology.orchestra.run.smithi150.stdout:Total 7.4 MB/s | 6.0 MB 00:00 2023-08-16T02:02:21.546 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:02:21.557 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:02:21.557 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:02:21.636 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:02:21.640 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:02:22.616 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:02:22.974 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-08-16T02:02:23.188 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-08-16T02:02:23.558 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-08-16T02:02:23.835 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-08-16T02:02:24.070 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5526.gb5259484.el8.noarch 5/5 2023-08-16T02:02:24.306 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5526.gb5259484.el8.noarch 5/5 2023-08-16T02:02:24.306 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5526.gb5259484.el8.noarch 1/5 2023-08-16T02:02:24.306 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-08-16T02:02:24.306 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-08-16T02:02:24.306 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-08-16T02:02:25.335 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-08-16T02:02:25.335 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:42 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:28.245 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:02:28.430 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.5.1-7.el8.noarch 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-08-16T02:02:28.432 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:28.433 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:28.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2023-08-16T02:02:28.891 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:29.733 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:46 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:31.112 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout: x86_64 2:18.0.0-5526.gb5259484.el8 ceph 861 k 2023-08-16T02:02:31.113 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2023-08-16T02:02:31.114 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-08-16T02:02:31.114 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:31.114 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:02:31.114 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:31.114 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2023-08-16T02:02:31.114 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:31.115 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 944 k 2023-08-16T02:02:31.115 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 3.0 M 2023-08-16T02:02:31.115 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:02:31.184 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-08-16T02:02:31.409 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-18.0.0-5526.gb5259484.el8.x86_ 2.9 MB/s | 861 kB 00:00 2023-08-16T02:02:31.410 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:02:31.410 INFO:teuthology.orchestra.run.smithi150.stdout:Total 3.1 MB/s | 944 kB 00:00 2023-08-16T02:02:31.421 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:02:31.443 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:02:31.443 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:02:31.519 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:02:31.522 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:02:31.712 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:02:31.977 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-08-16T02:02:32.126 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 2/2 2023-08-16T02:02:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 2/2 2023-08-16T02:02:32.459 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 1/2 2023-08-16T02:02:33.819 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-08-16T02:02:33.819 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:50 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:36.563 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:02:36.744 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:36.745 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:02:36.745 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-08-16T02:02:36.745 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:36.745 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:36.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-volume 2023-08-16T02:02:37.226 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:38.011 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:55 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:39.427 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:39.428 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:39.428 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2023-08-16T02:02:39.429 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:39.429 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:02:39.429 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:18.0.0-5526.gb5259484.el8 ceph-noarch 261 k 2023-08-16T02:02:39.429 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:39.430 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:02:39.430 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:39.430 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:02:39.431 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:39.431 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 261 k 2023-08-16T02:02:39.431 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.3 M 2023-08-16T02:02:39.431 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:02:39.816 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-volume-18.0.0-5526.gb5259484.el8.noarch.rp 672 kB/s | 261 kB 00:00 2023-08-16T02:02:39.817 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:02:39.817 INFO:teuthology.orchestra.run.smithi150.stdout:Total 670 kB/s | 261 kB 00:00 2023-08-16T02:02:39.818 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:02:39.828 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:02:39.828 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:02:39.845 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:02:39.848 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:02:40.018 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:02:40.244 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:02:40.605 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:02:41.627 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:18.0.0-5526.gb5259484.el8.noarch 1/1 2023-08-16T02:02:41.628 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:58 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:44.118 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:02:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:44.297 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:02:44.297 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:18.0.0-5526.gb5259484.el8.noarch 2023-08-16T02:02:44.298 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:44.298 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:44.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2023-08-16T02:02:44.789 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:45.616 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:02 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:46.997 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:46.997 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:46.998 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-08-16T02:02:46.998 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:46.998 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:02:46.999 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:18.0.0-5526.gb5259484.el8 ceph 139 k 2023-08-16T02:02:46.999 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:47.000 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:02:47.000 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:47.000 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:02:47.000 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:47.001 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 139 k 2023-08-16T02:02:47.001 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 513 k 2023-08-16T02:02:47.001 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:02:47.307 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-18.0.0-5526.gb5259484.el8.x86_64 451 kB/s | 139 kB 00:00 2023-08-16T02:02:47.307 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:02:47.308 INFO:teuthology.orchestra.run.smithi150.stdout:Total 449 kB/s | 139 kB 00:00 2023-08-16T02:02:47.308 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:02:47.314 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:02:47.314 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:02:47.367 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:02:47.371 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:02:47.482 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:02:47.664 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:02:47.901 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:02:48.953 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:02:48.953 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:05 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:51.522 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:02:51.704 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:51.704 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:02:51.704 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:02:51.704 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:51.704 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:51.832 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2023-08-16T02:02:52.189 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:53.115 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:10 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:54.420 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:02:54.528 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:54.528 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:02:54.529 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:02:54.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2023-08-16T02:02:54.986 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:02:56.597 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:13 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:02:57.988 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:02:57.989 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:57.989 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-08-16T02:02:57.989 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:57.989 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:02:57.990 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:18.0.0-5526.gb5259484.el8 ceph 31 k 2023-08-16T02:02:57.990 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:57.990 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:02:57.990 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:02:57.990 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:02:57.990 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:02:57.991 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 31 k 2023-08-16T02:02:57.991 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 140 k 2023-08-16T02:02:57.991 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:02:58.136 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-18.0.0-5526.gb5259484.el8.x86_6 213 kB/s | 31 kB 00:00 2023-08-16T02:02:58.137 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:02:58.137 INFO:teuthology.orchestra.run.smithi150.stdout:Total 211 kB/s | 31 kB 00:00 2023-08-16T02:02:58.137 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:02:58.141 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:02:58.141 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:02:58.151 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:02:58.155 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:02:58.217 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:02:58.396 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:02:58.640 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:02:59.627 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:02:59.627 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:16 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:03.549 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:03:03.730 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:03.730 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:03:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:03:03.731 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:03.731 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:03.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2023-08-16T02:03:04.217 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:05.106 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:22 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:06.454 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:03:06.563 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:06.564 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:03:06.564 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:06.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2023-08-16T02:03:07.042 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:08.274 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:25 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:09.542 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:03:09.650 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:09.652 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:03:09.652 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:09.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2023-08-16T02:03:10.124 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:10.922 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:27 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:12.202 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:03:12.310 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:12.311 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:03:12.311 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:12.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2023-08-16T02:03:12.776 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:13.732 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:30 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:15.016 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:03:15.123 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:15.124 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:03:15.124 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:15.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2023-08-16T02:03:15.587 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:16.391 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:33 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:17.670 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:03:17.782 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:17.784 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:03:17.784 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:17.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2023-08-16T02:03:18.248 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:19.068 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:36 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:20.353 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:18.0.0-5526.gb5259484.el8.x86_64 is already installed. 2023-08-16T02:03:20.461 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:20.463 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2023-08-16T02:03:20.463 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:20.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2023-08-16T02:03:20.931 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:21.790 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:38 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:23.173 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:23.174 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:23.174 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-08-16T02:03:23.174 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:23.174 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:18.0.0-5526.gb5259484.el8 ceph 94 k 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:23.175 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 94 k 2023-08-16T02:03:23.176 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 276 k 2023-08-16T02:03:23.176 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:03:23.347 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-18.0.0-5526.gb5259484.el8.x86_64.rpm 545 kB/s | 94 kB 00:00 2023-08-16T02:03:23.347 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:03:23.348 INFO:teuthology.orchestra.run.smithi150.stdout:Total 541 kB/s | 94 kB 00:00 2023-08-16T02:03:23.348 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:03:23.359 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:03:23.359 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:03:23.412 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:03:23.415 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:03:23.530 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:03:23.676 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:23.908 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:24.947 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:24.947 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:41 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:27.262 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:03:27.446 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:27.446 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:03:27.447 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:03:27.447 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:27.447 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:27.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2023-08-16T02:03:27.926 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:28.802 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:45 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:30.219 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:30.220 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:30.220 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2023-08-16T02:03:30.220 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:30.220 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:03:30.220 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:18.0.0-5526.gb5259484.el8 ceph 3.5 M 2023-08-16T02:03:30.221 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:30.221 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:03:30.221 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:30.221 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:03:30.221 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:30.222 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 3.5 M 2023-08-16T02:03:30.222 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 17 M 2023-08-16T02:03:30.222 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:03:30.775 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-18.0.0-5526.gb5259484.el8.x86_64.rpm 6.4 MB/s | 3.5 MB 00:00 2023-08-16T02:03:30.776 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:03:30.777 INFO:teuthology.orchestra.run.smithi150.stdout:Total 6.4 MB/s | 3.5 MB 00:00 2023-08-16T02:03:30.777 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:03:30.789 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:03:30.789 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:03:30.856 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:03:30.860 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:03:31.310 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:03:31.478 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:31.847 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:32.914 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:32.914 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:49 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:35.711 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:03:35.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:35.895 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:03:35.895 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:03:35.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:35.895 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:36.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2023-08-16T02:03:36.382 INFO:teuthology.orchestra.run.smithi150.stdout:Updating Subscription Management repositories. 2023-08-16T02:03:37.225 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:54 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:38.637 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2023-08-16T02:03:38.638 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:38.638 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2023-08-16T02:03:38.638 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:38.638 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2023-08-16T02:03:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:18.0.0-5526.gb5259484.el8 ceph 179 k 2023-08-16T02:03:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:38.639 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2023-08-16T02:03:38.639 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2023-08-16T02:03:38.639 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2023-08-16T02:03:38.639 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:38.640 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 179 k 2023-08-16T02:03:38.640 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 565 k 2023-08-16T02:03:38.640 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2023-08-16T02:03:38.828 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-18.0.0-5526.gb5259484.el8.x86_64.rpm 945 kB/s | 179 kB 00:00 2023-08-16T02:03:38.829 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2023-08-16T02:03:38.829 INFO:teuthology.orchestra.run.smithi150.stdout:Total 939 kB/s | 179 kB 00:00 2023-08-16T02:03:38.830 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2023-08-16T02:03:38.841 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2023-08-16T02:03:38.841 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2023-08-16T02:03:38.895 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2023-08-16T02:03:38.899 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2023-08-16T02:03:39.021 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2023-08-16T02:03:39.174 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:39.399 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:40.458 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:18.0.0-5526.gb5259484.el8.x86_64 1/1 2023-08-16T02:03:40.458 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:03:57 ago on Wed 16 Aug 2023 01:59:43 AM UTC. 2023-08-16T02:03:44.199 INFO:teuthology.orchestra.run.smithi150.stdout:Installed products updated. 2023-08-16T02:03:44.379 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:44.380 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2023-08-16T02:03:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:18.0.0-5526.gb5259484.el8.x86_64 2023-08-16T02:03:44.380 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-08-16T02:03:44.380 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2023-08-16T02:03:44.509 DEBUG:teuthology.parallel:result is None 2023-08-16T02:03:44.510 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b5259484dbc61e8573f27e692560c6c107b9d4c1 2023-08-16T02:03:44.662 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-16T02:03:44.684 INFO:teuthology.orchestra.run.smithi150.stdout:18.0.0-5526.gb5259484.el8 2023-08-16T02:03:44.685 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5526.gb5259484.el8 2023-08-16T02:03:44.685 INFO:teuthology.task.install:The correct ceph version 18.0.0-5526.gb5259484 is installed. 2023-08-16T02:03:44.685 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-08-16T02:03:44.686 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T02:03:44.686 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-16T02:03:44.753 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-08-16T02:03:44.753 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T02:03:44.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2023-08-16T02:03:44.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-16T02:03:44.887 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-08-16T02:03:44.887 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T02:03:44.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-16T02:03:44.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-16T02:03:45.020 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-08-16T02:03:45.020 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T02:03:45.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2023-08-16T02:03:45.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-16T02:03:45.153 INFO:teuthology.run_tasks:Running task workunit... 2023-08-16T02:03:45.168 INFO:tasks.workunit:Pulling workunits from ref b5259484dbc61e8573f27e692560c6c107b9d4c1 2023-08-16T02:03:45.168 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-08-16T02:03:45.169 INFO:tasks.workunit:timeout=3h 2023-08-16T02:03:45.169 INFO:tasks.workunit:cleanup=True 2023-08-16T02:03:45.169 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.0 2023-08-16T02:03:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T02:03:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-08-16T02:03:45.209 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-08-16T02:03:45.266 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-08-16T02:03:45.267 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-08-16T02:03:45.322 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b5259484dbc61e8573f27e692560c6c107b9d4c1 2023-08-16T02:03:45.379 INFO:tasks.workunit.client.0.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-08-16T02:04:27.080 INFO:tasks.workunit.client.0.smithi150.stderr:Note: switching to 'b5259484dbc61e8573f27e692560c6c107b9d4c1'. 2023-08-16T02:04:27.080 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.080 INFO:tasks.workunit.client.0.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-08-16T02:04:27.080 INFO:tasks.workunit.client.0.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr:state without impacting any branches by switching back to a branch. 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr: git switch -c 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.081 INFO:tasks.workunit.client.0.smithi150.stderr:Or undo this operation with: 2023-08-16T02:04:27.082 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.082 INFO:tasks.workunit.client.0.smithi150.stderr: git switch - 2023-08-16T02:04:27.082 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.082 INFO:tasks.workunit.client.0.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-08-16T02:04:27.082 INFO:tasks.workunit.client.0.smithi150.stderr: 2023-08-16T02:04:27.082 INFO:tasks.workunit.client.0.smithi150.stderr:HEAD is now at b5259484dbc Merge branch 'wip-ec-static_encode_and_write' of https://github.com/rzarzynski/ceph into wip-yuri3-testing-2023-08-15-0955 2023-08-16T02:04:27.083 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-08-16T02:04:27.141 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2023-08-16T02:04:27.141 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-08-16T02:04:27.198 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2023-08-16T02:04:27.199 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2023-08-16T02:04:27.199 DEBUG:teuthology.orchestra.run.smithi150:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5259484dbc61e8573f27e692560c6c107b9d4c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2023-08-16T02:04:27.276 INFO:tasks.workunit.client.0.smithi150.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-08-16T02:04:27.284 INFO:tasks.workunit.client.0.smithi150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-08-16T02:04:27.285 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-08-16T02:04:27.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-08-16T02:04:27.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-08-16T02:04:27.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-08-16T02:04:27.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-08-16T02:04:27.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-08-16T02:04:27.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-08-16T02:04:27.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-08-16T02:04:27.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-mapper 2023-08-16T02:04:27.288 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2023-08-16T02:04:27.288 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2023-08-16T02:04:27.288 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2023-08-16T02:04:27.288 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2023-08-16T02:04:27.289 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2023-08-16T02:04:27.289 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2023-08-16T02:04:27.289 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none ' 2023-08-16T02:04:27.289 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2023-08-16T02:04:27.290 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-08-16T02:04:27.290 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2023-08-16T02:04:27.290 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-08-16T02:04:27.295 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2023-08-16T02:04:27.295 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2023-08-16T02:04:27.296 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2023-08-16T02:04:27.296 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2023-08-16T02:04:27.296 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2023-08-16T02:04:27.296 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-08-16T02:04:27.297 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-08-16T02:04:27.297 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-08-16T02:04:27.297 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:04:27.298 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:04:27.298 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:04:27.298 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:04:27.298 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:04:27.299 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:04:27.299 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:04:27.299 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:04:27.299 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:04:27.300 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:04:27.300 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:04:27.300 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:04:27.301 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:04:27.301 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:04:27.301 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:04:27.301 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:04:27.302 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:04:27.302 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-08-16T02:04:27.302 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-08-16T02:04:27.302 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:04:27.303 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:04:27.303 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:04:27.303 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93296 2023-08-16T02:04:27.306 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:04:27.307 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:04:27.307 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2023-08-16T02:04:27.307 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-08-16T02:04:27.308 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:04:27.308 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:04:27.308 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.93296 2023-08-16T02:04:27.308 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-08-16T02:04:27.309 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-08-16T02:04:27.309 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-08-16T02:04:27.309 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-08-16T02:04:27.310 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2023-08-16T02:04:27.310 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2023-08-16T02:04:27.310 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2023-08-16T02:04:27.310 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']="3" ['pgs_in_pool']="4" ['pool_name']="test") 2023-08-16T02:04:27.311 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2023-08-16T02:04:27.311 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2023-08-16T02:04:27.311 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2023-08-16T02:04:27.312 INFO:tasks.workunit.client.0.smithi150.stdout:Dir: td/osd-mapper 2023-08-16T02:04:27.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2023-08-16T02:04:27.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2023-08-16T02:04:27.315 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2023-08-16T02:04:27.315 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2023-08-16T02:04:27.315 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2023-08-16T02:04:27.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2023-08-16T02:04:27.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2023-08-16T02:04:27.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2023-08-16T02:04:27.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2023-08-16T02:04:27.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2023-08-16T02:04:27.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2023-08-16T02:04:27.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2023-08-16T02:04:29.085 INFO:tasks.workunit.client.0.smithi150.stdout:add osd0 d0aeff71-f436-44c5-8302-df026022ff38 2023-08-16T02:04:29.372 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:04:29.414 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.411+0000 7f693cceb640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:04:29.431 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.429+0000 7f693cceb640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:04:29.432 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.430+0000 7f693cceb640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:04:29.433 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.430+0000 7f693cceb640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-08-16T02:04:29.433 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.430+0000 7f693cceb640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-08-16T02:04:29.433 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.430+0000 7f693cceb640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2023-08-16T02:04:29.515 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:29.513+0000 7f693cceb640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2023-08-16T02:04:34.991 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd0 key to auth repository 2023-08-16T02:04:35.185 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.0 2023-08-16T02:04:35.214 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:35.211+0000 7fcf31ad8640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:04:35.216 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:35.214+0000 7fcf31ad8640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:04:35.218 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:35.215+0000 7fcf31ad8640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:04:35.378 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:04:35.745 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:35.743+0000 7fcf31ad8640 -1 Falling back to public interface 2023-08-16T02:04:36.591 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:04:37.901 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:04:38.213 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:04:38.211+0000 7fcf31ad8640 -1 osd.0 0 log_to_monitors true 2023-08-16T02:04:39.221 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:04:40.541 INFO:tasks.workunit.client.0.smithi150.stdout:4 2023-08-16T02:04:41.862 INFO:tasks.workunit.client.0.smithi150.stdout:5 2023-08-16T02:04:43.193 INFO:tasks.workunit.client.0.smithi150.stdout:6 2023-08-16T02:04:44.517 INFO:tasks.workunit.client.0.smithi150.stdout:7 2023-08-16T02:04:45.840 INFO:tasks.workunit.client.0.smithi150.stdout:8 2023-08-16T02:04:47.189 INFO:tasks.workunit.client.0.smithi150.stdout:9 2023-08-16T02:04:48.513 INFO:tasks.workunit.client.0.smithi150.stdout:10 2023-08-16T02:04:49.839 INFO:tasks.workunit.client.0.smithi150.stdout:11 2023-08-16T02:04:51.495 INFO:tasks.workunit.client.0.smithi150.stdout:12 2023-08-16T02:04:52.980 INFO:tasks.workunit.client.0.smithi150.stdout:13 2023-08-16T02:04:54.650 INFO:tasks.workunit.client.0.smithi150.stdout:14 2023-08-16T02:04:56.326 INFO:tasks.workunit.client.0.smithi150.stdout:15 2023-08-16T02:04:57.705 INFO:tasks.workunit.client.0.smithi150.stdout:16 2023-08-16T02:04:59.410 INFO:tasks.workunit.client.0.smithi150.stdout:17 2023-08-16T02:05:01.109 INFO:tasks.workunit.client.0.smithi150.stdout:18 2023-08-16T02:05:03.223 INFO:tasks.workunit.client.0.smithi150.stdout:19 2023-08-16T02:05:03.541 INFO:tasks.workunit.client.0.smithi150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4017085710,v1:127.0.0.1:6801/4017085710] [v2:127.0.0.1:6802/4017085710,v1:127.0.0.1:6803/4017085710] exists,up d0aeff71-f436-44c5-8302-df026022ff38 2023-08-16T02:05:03.544 INFO:tasks.workunit.client.0.smithi150.stdout:add osd1 069972a8-6def-4578-a146-b36c6299a3cc 2023-08-16T02:05:03.947 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:05:03.990 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:03.988+0000 7f66c7e81640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:04.048 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:04.045+0000 7f66c7e81640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:04.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:04.046+0000 7f66c7e81640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:04.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:04.046+0000 7f66c7e81640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-08-16T02:05:04.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:04.046+0000 7f66c7e81640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-08-16T02:05:04.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:04.046+0000 7f66c7e81640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2023-08-16T02:05:04.119 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:04.116+0000 7f66c7e81640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2023-08-16T02:05:09.882 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd1 key to auth repository 2023-08-16T02:05:10.282 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.1 2023-08-16T02:05:10.310 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:10.307+0000 7f05c95a2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:10.321 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:10.318+0000 7f05c95a2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:10.321 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:10.319+0000 7f05c95a2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:10.609 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:05:10.848 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:10.846+0000 7f05c95a2640 -1 Falling back to public interface 2023-08-16T02:05:11.955 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:05:13.073 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:13.071+0000 7f05c95a2640 -1 osd.1 0 log_to_monitors true 2023-08-16T02:05:13.325 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:05:14.763 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:05:16.082 INFO:tasks.workunit.client.0.smithi150.stdout:4 2023-08-16T02:05:17.496 INFO:tasks.workunit.client.0.smithi150.stdout:5 2023-08-16T02:05:18.873 INFO:tasks.workunit.client.0.smithi150.stdout:6 2023-08-16T02:05:20.193 INFO:tasks.workunit.client.0.smithi150.stdout:7 2023-08-16T02:05:21.516 INFO:tasks.workunit.client.0.smithi150.stdout:8 2023-08-16T02:05:22.863 INFO:tasks.workunit.client.0.smithi150.stdout:9 2023-08-16T02:05:24.181 INFO:tasks.workunit.client.0.smithi150.stdout:10 2023-08-16T02:05:25.505 INFO:tasks.workunit.client.0.smithi150.stdout:11 2023-08-16T02:05:27.108 INFO:tasks.workunit.client.0.smithi150.stdout:12 2023-08-16T02:05:28.517 INFO:tasks.workunit.client.0.smithi150.stdout:13 2023-08-16T02:05:30.341 INFO:tasks.workunit.client.0.smithi150.stdout:14 2023-08-16T02:05:31.934 INFO:tasks.workunit.client.0.smithi150.stdout:15 2023-08-16T02:05:33.861 INFO:tasks.workunit.client.0.smithi150.stdout:16 2023-08-16T02:05:35.462 INFO:tasks.workunit.client.0.smithi150.stdout:17 2023-08-16T02:05:35.854 INFO:tasks.workunit.client.0.smithi150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1702997618,v1:127.0.0.1:6811/1702997618] [v2:127.0.0.1:6812/1702997618,v1:127.0.0.1:6813/1702997618] exists,up 069972a8-6def-4578-a146-b36c6299a3cc 2023-08-16T02:05:35.857 INFO:tasks.workunit.client.0.smithi150.stdout:add osd2 bd941c8c-5bea-4e6e-83bb-992681b7c40a 2023-08-16T02:05:36.354 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:05:36.397 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.395+0000 7fb428c07640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:36.438 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.436+0000 7fb428c07640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:36.439 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.438+0000 7fb428c07640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:36.440 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.438+0000 7fb428c07640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-08-16T02:05:36.440 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.438+0000 7fb428c07640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-08-16T02:05:36.440 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.438+0000 7fb428c07640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2023-08-16T02:05:36.521 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:36.520+0000 7fb428c07640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2023-08-16T02:05:42.234 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd2 key to auth repository 2023-08-16T02:05:42.640 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.2 2023-08-16T02:05:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:42.666+0000 7f4d40922640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:42.669+0000 7f4d40922640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:42.670+0000 7f4d40922640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:05:42.951 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:05:43.191 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:43.190+0000 7f4d40922640 -1 Falling back to public interface 2023-08-16T02:05:44.295 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:05:45.476 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:05:45.474+0000 7f4d40922640 -1 osd.2 0 log_to_monitors true 2023-08-16T02:05:45.616 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:05:46.998 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:05:48.322 INFO:tasks.workunit.client.0.smithi150.stdout:4 2023-08-16T02:05:49.678 INFO:tasks.workunit.client.0.smithi150.stdout:5 2023-08-16T02:05:51.092 INFO:tasks.workunit.client.0.smithi150.stdout:6 2023-08-16T02:05:52.444 INFO:tasks.workunit.client.0.smithi150.stdout:7 2023-08-16T02:05:53.879 INFO:tasks.workunit.client.0.smithi150.stdout:8 2023-08-16T02:05:55.326 INFO:tasks.workunit.client.0.smithi150.stdout:9 2023-08-16T02:05:56.650 INFO:tasks.workunit.client.0.smithi150.stdout:10 2023-08-16T02:05:57.974 INFO:tasks.workunit.client.0.smithi150.stdout:11 2023-08-16T02:05:59.639 INFO:tasks.workunit.client.0.smithi150.stdout:12 2023-08-16T02:06:01.219 INFO:tasks.workunit.client.0.smithi150.stdout:13 2023-08-16T02:06:02.749 INFO:tasks.workunit.client.0.smithi150.stdout:14 2023-08-16T02:06:04.094 INFO:tasks.workunit.client.0.smithi150.stdout:15 2023-08-16T02:06:05.924 INFO:tasks.workunit.client.0.smithi150.stdout:16 2023-08-16T02:06:07.810 INFO:tasks.workunit.client.0.smithi150.stdout:17 2023-08-16T02:06:09.189 INFO:tasks.workunit.client.0.smithi150.stdout:18 2023-08-16T02:06:09.512 INFO:tasks.workunit.client.0.smithi150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2415791190,v1:127.0.0.1:6819/2415791190] [v2:127.0.0.1:6820/2415791190,v1:127.0.0.1:6821/2415791190] exists,up bd941c8c-5bea-4e6e-83bb-992681b7c40a 2023-08-16T02:06:10.108 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'test' created 2023-08-16T02:06:12.067 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.0 seq 21474836501 2023-08-16T02:06:12.400 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.1 seq 42949672981 2023-08-16T02:06:12.753 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.2 seq 64424509450 2023-08-16T02:06:16.906 INFO:tasks.workunit.client.0.smithi150.stdout:standard_scrub_cluster: dbg: test pool is test 1 2023-08-16T02:06:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2023-08-16T02:06:17.021 INFO:tasks.workunit.client.0.smithi150.stdout:osd.0: { 2023-08-16T02:06:17.022 INFO:tasks.workunit.client.0.smithi150.stdout: "success": "osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-08-16T02:06:17.022 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:17.037 INFO:tasks.workunit.client.0.smithi150.stdout:osd.1: { 2023-08-16T02:06:17.037 INFO:tasks.workunit.client.0.smithi150.stdout: "success": "osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-08-16T02:06:17.037 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:17.052 INFO:tasks.workunit.client.0.smithi150.stdout:osd.2: { 2023-08-16T02:06:17.052 INFO:tasks.workunit.client.0.smithi150.stdout: "success": "osd_stats_update_period_not_scrubbing = '1' (not observed, change may require restart) " 2023-08-16T02:06:17.053 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:17.063 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2023-08-16T02:06:17.178 INFO:tasks.workunit.client.0.smithi150.stdout:osd.0: { 2023-08-16T02:06:17.179 INFO:tasks.workunit.client.0.smithi150.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-08-16T02:06:17.179 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:17.194 INFO:tasks.workunit.client.0.smithi150.stdout:osd.1: { 2023-08-16T02:06:17.194 INFO:tasks.workunit.client.0.smithi150.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-08-16T02:06:17.195 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:17.209 INFO:tasks.workunit.client.0.smithi150.stdout:osd.2: { 2023-08-16T02:06:17.209 INFO:tasks.workunit.client.0.smithi150.stdout: "success": "osd_stats_update_period_scrubbing = '1' (not observed, change may require restart) " 2023-08-16T02:06:17.209 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:17.220 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2023-08-16T02:06:17.220 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2023-08-16T02:06:17.220 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2023-08-16T02:06:17.220 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2023-08-16T02:06:17.220 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2023-08-16T02:06:17.221 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-08-16T02:06:17.221 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-08-16T02:06:17.535 INFO:tasks.workunit.client.0.smithi150.stdout:created pool test snap snap01 2023-08-16T02:06:17.743 INFO:tasks.workunit.client.0.smithi150.stdout:created pool test snap snap02 2023-08-16T02:06:17.745 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2023-08-16T02:06:17.745 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-08-16T02:06:17.745 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-08-16T02:06:18.312 INFO:tasks.workunit.client.0.smithi150.stdout:created pool test snap snap13 2023-08-16T02:06:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2023-08-16T02:06:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2023-08-16T02:06:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2023-08-16T02:06:18.783 INFO:tasks.workunit.client.0.smithi150.stdout:created pool test snap snap24 2023-08-16T02:06:18.974 INFO:tasks.workunit.client.0.smithi150.stdout:created pool test snap snap25 2023-08-16T02:06:18.977 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 19378 2023-08-16T02:06:18.977 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2023-08-16T02:06:19.230 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-08-16T02:06:19.231 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2023-08-16T02:06:19.549 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2023-08-16T02:06:19.550 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-08-16T02:06:19.550 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2023-08-16T02:06:19.900 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2023-08-16T02:06:19.900 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2023-08-16T02:06:19.901 INFO:tasks.workunit.client.0.smithi150.stdout:pgid is 1.3 (primary: osd.1) 2023-08-16T02:06:19.902 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2023-08-16T02:06:19.902 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:294: set_query_debug: local pgid=1.3 2023-08-16T02:06:19.902 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: ceph pg dump pgs_brief 2023-08-16T02:06:19.902 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2023-08-16T02:06:20.259 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs_brief 2023-08-16T02:06:20.269 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: local prim_osd=1 2023-08-16T02:06:20.270 INFO:tasks.workunit.client.0.smithi150.stdout:Setting scrub debug data. Primary for 1.3 is 1 2023-08-16T02:06:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2023-08-16T02:06:20.271 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: get_asok_path osd.1 2023-08-16T02:06:20.271 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2023-08-16T02:06:20.271 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2023-08-16T02:06:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-08-16T02:06:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:06:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:06:20.272 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93296/ceph-osd.1.asok 2023-08-16T02:06:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: CEPH_ARGS= 2023-08-16T02:06:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.93296/ceph-osd.1.asok scrubdebug 1.3 set sessions 2023-08-16T02:06:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-08-16T02:06:20.415 INFO:tasks.workunit.client.0.smithi150.stdout:{"success":true}{ 2023-08-16T02:06:20.415 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "objxxx", 2023-08-16T02:06:20.415 INFO:tasks.workunit.client.0.smithi150.stdout: "clones": [ 2023-08-16T02:06:20.415 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.415 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2023-08-16T02:06:20.415 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [ 2023-08-16T02:06:20.416 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.416 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 1, 2023-08-16T02:06:20.416 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap01" 2023-08-16T02:06:20.416 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:06:20.416 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap02" 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 5, 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: "overlaps": [] 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:06:20.417 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [ 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap13" 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:06:20.418 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 6, 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: "overlaps": [] 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [ 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.419 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 4, 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap24" 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap25" 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2023-08-16T02:06:20.420 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 6, 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: "overlaps": [] 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: "id": "head", 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [], 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 6 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:06:20.421 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2023-08-16T02:06:20.422 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:20.422 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-08-16T02:06:20.532 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:06:20.532 INFO:tasks.workunit.client.0.smithi150.stdout: "deep": true, 2023-08-16T02:06:20.532 INFO:tasks.workunit.client.0.smithi150.stdout: "stamp": "2023-08-09T02:04:40.531296+0000" 2023-08-16T02:06:20.533 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:06:20.542 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2023-08-16T02:06:23.544 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2023-08-16T02:06:23.844 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:06:23.844 INFO:tasks.workunit.client.0.smithi150.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2023-08-16T02:06:21.063414+0000 27'7 27:43 [1,2,0] 1 [1,2,0] 1 27'7 2023-08-16T02:06:21.063288+0000 27'7 2023-08-16T02:06:21.063288+0000 0 1 periodic scrub scheduled @ 2023-08-17T02:06:21.063288+0000 4 0 2023-08-16T02:06:23.844 INFO:tasks.workunit.client.0.smithi150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:06:11.152542+0000 0'0 22:17 [0,1,2] 0 [0,1,2] 0 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:06:23.845 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:06:11.185943+0000 0'0 27:26 [1,0,2] 1 [1,0,2] 1 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:06:23.845 INFO:tasks.workunit.client.0.smithi150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:06:11.068640+0000 0'0 26:22 [2,0,1] 2 [2,0,1] 2 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:06:23.845 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:23.845 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:06:23.847 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:06:23.856 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2023-08-16T02:06:23.858 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2023-08-16T02:06:24.146 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:06:24.147 INFO:tasks.workunit.client.0.smithi150.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2023-08-16T02:06:21.063414+0000 27'7 27:43 [1,2,0] 1 [1,2,0] 1 27'7 2023-08-16T02:06:21.063288+0000 27'7 2023-08-16T02:06:21.063288+0000 0 1 periodic scrub scheduled @ 2023-08-17T02:06:21.063288+0000 4 0 2023-08-16T02:06:24.147 INFO:tasks.workunit.client.0.smithi150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:06:11.152542+0000 0'0 27:26 [0,1,2] 0 [0,1,2] 0 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:06:24.147 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:06:11.185943+0000 0'0 27:26 [1,0,2] 1 [1,0,2] 1 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:06:24.147 INFO:tasks.workunit.client.0.smithi150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:06:11.068640+0000 0'0 27:25 [2,0,1] 2 [2,0,1] 2 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:06:24.147 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:24.147 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:06:24.149 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:06:24.158 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2023-08-16T02:06:24.558 INFO:tasks.workunit.client.0.smithi150.stderr:noscrub is set 2023-08-16T02:06:24.568 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2023-08-16T02:06:24.994 INFO:tasks.workunit.client.0.smithi150.stderr:nodeep-scrub is set 2023-08-16T02:06:25.004 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2023-08-16T02:06:30.005 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2023-08-16T02:06:30.006 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-08-16T02:06:30.007 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:06:30.007 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:06:30.008 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:06:30.008 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:06:30.008 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:06:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:06:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-08-16T02:06:30.342 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.340+0000 7f91965d1980 1 bdev(0x56022764a000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-08-16T02:06:30.343 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.340+0000 7f91965d1980 1 bdev(0x56022764a000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:30.343 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.340+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-08-16T02:06:30.343 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.341+0000 7f91965d1980 1 bdev(0x56022764a380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-08-16T02:06:30.343 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.341+0000 7f91965d1980 1 bdev(0x56022764a380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:30.343 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.341+0000 7f91965d1980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-08-16T02:06:30.343 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.341+0000 7f91965d1980 1 bluefs mount 2023-08-16T02:06:30.347 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.345+0000 7f91965d1980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:30.356 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.354+0000 7f91965d1980 1 bluefs mount shared_bdev_used = 0 2023-08-16T02:06:30.356 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.354+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:30.358 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.356+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.356+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.356+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.356+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.356+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:30.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:30.360 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:30.360 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:30.360 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.357+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:30.390 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2023-08-16T02:06:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-08-16T02:06:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-08-16T02:06:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-08-16T02:06:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 freelist init 2023-08-16T02:06:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.389+0000 7f91965d1980 1 freelist _read_cfg 2023-08-16T02:06:30.392 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.390+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 21 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 7.62941e-07 2023-08-16T02:06:30.392 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.390+0000 7f91965d1980 1 bluefs umount 2023-08-16T02:06:30.392 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.390+0000 7f91965d1980 1 bdev(0x56022764a380 td/osd-mapper/0/block) close 2023-08-16T02:06:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.606+0000 7f91965d1980 1 bdev(0x56022764a380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-08-16T02:06:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.606+0000 7f91965d1980 1 bdev(0x56022764a380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.606+0000 7f91965d1980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-08-16T02:06:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.606+0000 7f91965d1980 1 bluefs mount 2023-08-16T02:06:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.620+0000 7f91965d1980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:30.630 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.628+0000 7f91965d1980 1 bluefs mount shared_bdev_used = 27459584 2023-08-16T02:06:30.630 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.628+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:30.633 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.630+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:30.633 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:30.633 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:30.633 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:30.633 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:30.631+0000 7f91965d1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:31.359 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.357+0000 7f91965d1980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:31.361 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-08-16T02:06:31.361 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 0a 05 4c 02 00 00 67 72 0a da c5 4a 49 f5 9d ac |..L...gr...JI...| 2023-08-16T02:06:31.361 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 47 ee b7 a9 ba 6d 00 00 00 00 1d 00 00 00 01 00 |G....m..........| 2023-08-16T02:06:31.361 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:31.361 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:31.361 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-08-16T02:06:31.362 INFO:tasks.workunit.client.0.smithi150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-08-16T02:06:31.363 INFO:tasks.workunit.client.0.smithi150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-08-16T02:06:31.364 INFO:tasks.workunit.client.0.smithi150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-08-16T02:06:31.365 INFO:tasks.workunit.client.0.smithi150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout:00000220 72 65 1d 00 00 00 05 00 00 00 d0 ae ff 71 f4 36 |re...........q.6| 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout:00000230 44 c5 83 02 df 02 60 22 ff 38 00 00 00 00 00 00 |D.....`".8......| 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout:00000240 00 00 1d 00 00 00 ce 2e dc 64 7e a2 fa 03 01 00 |.........d~.....| 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout:00000250 00 00 |..| 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout:00000252 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-08-16T02:06:31.366 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:31.367 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout:0000003f 2023-08-16T02:06:31.368 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:31.369 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:31.370 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.371 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:31.372 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:31.373 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.374 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-08-16T02:06:31.375 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:31.375 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.358+0000 7f91965d1980 1 bluefs umount 2023-08-16T02:06:31.375 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.358+0000 7f91965d1980 1 bdev(0x56022764a380 td/osd-mapper/0/block) close 2023-08-16T02:06:31.598 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.597+0000 7f91965d1980 1 freelist shutdown 2023-08-16T02:06:31.599 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.597+0000 7f91965d1980 1 bdev(0x56022764a000 td/osd-mapper/0/block) close 2023-08-16T02:06:31.861 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:31.862 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-08-16T02:06:31.862 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2023-08-16T02:06:31.887 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.885+0000 7feee9120980 1 bdev(0x555f2c460000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-08-16T02:06:31.887 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.885+0000 7feee9120980 1 bdev(0x555f2c460000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:31.888 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.885+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-08-16T02:06:31.888 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.885+0000 7feee9120980 1 bdev(0x555f2c460380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-08-16T02:06:31.888 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.886+0000 7feee9120980 1 bdev(0x555f2c460380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:31.888 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.886+0000 7feee9120980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-08-16T02:06:31.888 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.886+0000 7feee9120980 1 bluefs mount 2023-08-16T02:06:31.904 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.902+0000 7feee9120980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:31.912 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.911+0000 7feee9120980 1 bluefs mount shared_bdev_used = 0 2023-08-16T02:06:31.913 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.911+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:31.916 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:31.917 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:31.917 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:31.917 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:31.917 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:31.917 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.913+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-08-16T02:06:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-08-16T02:06:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-08-16T02:06:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-08-16T02:06:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 freelist init 2023-08-16T02:06:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.945+0000 7feee9120980 1 freelist _read_cfg 2023-08-16T02:06:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.946+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 12 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 4.19618e-07 2023-08-16T02:06:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.946+0000 7feee9120980 1 bluefs umount 2023-08-16T02:06:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:31.946+0000 7feee9120980 1 bdev(0x555f2c460380 td/osd-mapper/2/block) close 2023-08-16T02:06:32.156 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.155+0000 7feee9120980 1 bdev(0x555f2c460380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-08-16T02:06:32.157 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.155+0000 7feee9120980 1 bdev(0x555f2c460380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:32.157 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.155+0000 7feee9120980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-08-16T02:06:32.157 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.155+0000 7feee9120980 1 bluefs mount 2023-08-16T02:06:32.167 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.166+0000 7feee9120980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.179+0000 7feee9120980 1 bluefs mount shared_bdev_used = 27459584 2023-08-16T02:06:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.179+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:32.183 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:32.184 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:32.184 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:32.184 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.181+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.182+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.182+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:32.186 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.182+0000 7feee9120980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:32.963 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.961+0000 7feee9120980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:32.964 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.962+0000 7feee9120980 1 bluefs umount 2023-08-16T02:06:32.964 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:32.962+0000 7feee9120980 1 bdev(0x555f2c460380 td/osd-mapper/2/block) close 2023-08-16T02:06:33.202 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:33.201+0000 7feee9120980 1 freelist shutdown 2023-08-16T02:06:33.203 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:33.201+0000 7feee9120980 1 bdev(0x555f2c460000 td/osd-mapper/2/block) close 2023-08-16T02:06:33.405 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:33.406 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:33.406 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:33.406 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:33.406 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:33.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:33.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2023-08-16T02:06:33.407 INFO:tasks.workunit.client.0.smithi150.stderr:grep: /tmp/oo2.dump: No such file or directory 2023-08-16T02:06:33.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:33.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-08-16T02:06:33.941 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-08-16T02:06:33.941 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 0a 05 4c 02 00 00 67 72 0a da c5 4a 49 f5 9d ac |..L...gr...JI...| 2023-08-16T02:06:33.942 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 47 ee b7 a9 ba 6d 02 00 00 00 1d 00 00 00 01 00 |G....m..........| 2023-08-16T02:06:33.942 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:33.942 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:33.942 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-08-16T02:06:33.943 INFO:tasks.workunit.client.0.smithi150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-08-16T02:06:33.943 INFO:tasks.workunit.client.0.smithi150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-08-16T02:06:33.943 INFO:tasks.workunit.client.0.smithi150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-08-16T02:06:33.943 INFO:tasks.workunit.client.0.smithi150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-08-16T02:06:33.943 INFO:tasks.workunit.client.0.smithi150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-08-16T02:06:33.944 INFO:tasks.workunit.client.0.smithi150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-08-16T02:06:33.944 INFO:tasks.workunit.client.0.smithi150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-08-16T02:06:33.944 INFO:tasks.workunit.client.0.smithi150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-08-16T02:06:33.944 INFO:tasks.workunit.client.0.smithi150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-08-16T02:06:33.945 INFO:tasks.workunit.client.0.smithi150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-08-16T02:06:33.945 INFO:tasks.workunit.client.0.smithi150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-08-16T02:06:33.945 INFO:tasks.workunit.client.0.smithi150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-08-16T02:06:33.946 INFO:tasks.workunit.client.0.smithi150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-08-16T02:06:33.946 INFO:tasks.workunit.client.0.smithi150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-08-16T02:06:33.946 INFO:tasks.workunit.client.0.smithi150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-08-16T02:06:33.946 INFO:tasks.workunit.client.0.smithi150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-08-16T02:06:33.946 INFO:tasks.workunit.client.0.smithi150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-08-16T02:06:33.946 INFO:tasks.workunit.client.0.smithi150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-08-16T02:06:33.947 INFO:tasks.workunit.client.0.smithi150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-08-16T02:06:33.947 INFO:tasks.workunit.client.0.smithi150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-08-16T02:06:33.947 INFO:tasks.workunit.client.0.smithi150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-08-16T02:06:33.947 INFO:tasks.workunit.client.0.smithi150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-08-16T02:06:33.947 INFO:tasks.workunit.client.0.smithi150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-08-16T02:06:33.948 INFO:tasks.workunit.client.0.smithi150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-08-16T02:06:33.948 INFO:tasks.workunit.client.0.smithi150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-08-16T02:06:33.948 INFO:tasks.workunit.client.0.smithi150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-08-16T02:06:33.948 INFO:tasks.workunit.client.0.smithi150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-08-16T02:06:33.949 INFO:tasks.workunit.client.0.smithi150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-08-16T02:06:33.949 INFO:tasks.workunit.client.0.smithi150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-08-16T02:06:33.949 INFO:tasks.workunit.client.0.smithi150.stdout:00000220 72 65 1d 00 00 00 0f 00 00 00 bd 94 1c 8c 5b ea |re............[.| 2023-08-16T02:06:33.949 INFO:tasks.workunit.client.0.smithi150.stdout:00000230 4e 6e 83 bb 99 26 81 b7 c4 0a 00 00 00 00 00 00 |Nn...&..........| 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout:00000240 00 00 1d 00 00 00 0f 2f dc 64 e4 12 03 38 01 00 |......./.d...8..| 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout:00000250 00 00 |..| 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout:00000252 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.950 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:33.951 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:33.951 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:33.951 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:33.951 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-08-16T02:06:33.951 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:33.952 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:33.952 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.952 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:33.952 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:0000003f 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:33.954 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:33.954 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:33.954 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-08-16T02:06:33.954 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:33.955 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:33.955 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.955 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:33.955 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:33.955 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:33.956 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:33.956 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:33.956 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:33.956 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:33.957 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:33.958 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.959 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:33.960 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:33.960 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:33.960 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:33.960 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:33.960 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:33.961 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.961 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-08-16T02:06:33.961 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:33.961 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-08-16T02:06:33.961 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:34.206 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-08-16T02:06:34.206 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-08-16T02:06:34.206 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-08-16T02:06:34.207 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2023-08-16T02:06:35.755 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2023-08-16T02:06:35.755 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2023-08-16T02:06:35.757 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2023-08-16T02:06:35.758 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:06:35.758 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2023-08-16T02:06:35.758 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2023-08-16T02:06:35.758 INFO:tasks.workunit.client.0.smithi150.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2023-08-16T02:06:35.759 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-08-16T02:06:35.759 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-08-16T02:06:35.784 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bdev(0x5560cb15e000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-08-16T02:06:35.784 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bdev(0x5560cb15e000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:35.784 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-08-16T02:06:35.784 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bdev(0x5560cb15e380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-08-16T02:06:35.784 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bdev(0x5560cb15e380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:35.785 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-08-16T02:06:35.785 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.782+0000 7f458c5df980 1 bluefs mount 2023-08-16T02:06:35.788 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.786+0000 7f458c5df980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:35.797 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.796+0000 7f458c5df980 1 bluefs mount shared_bdev_used = 0 2023-08-16T02:06:35.798 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.796+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:35.800 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:35.800 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:35.800 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.798+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:35.801 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.799+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:35.802 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.799+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:35.802 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.799+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:35.805 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:35.805 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2023-08-16T02:06:35.806 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2023-08-16T02:06:35.806 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-08-16T02:06:35.806 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2023-08-16T02:06:35.806 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 freelist init 2023-08-16T02:06:35.806 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.804+0000 7f458c5df980 1 freelist _read_cfg 2023-08-16T02:06:35.807 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.805+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 21 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 7.62941e-07 2023-08-16T02:06:35.807 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.805+0000 7f458c5df980 1 bluefs umount 2023-08-16T02:06:35.807 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:35.805+0000 7f458c5df980 1 bdev(0x5560cb15e380 td/osd-mapper/0/block) close 2023-08-16T02:06:36.055 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.054+0000 7f458c5df980 1 bdev(0x5560cb15e380 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2023-08-16T02:06:36.056 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.054+0000 7f458c5df980 1 bdev(0x5560cb15e380 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:36.056 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.054+0000 7f458c5df980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2023-08-16T02:06:36.056 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.054+0000 7f458c5df980 1 bluefs mount 2023-08-16T02:06:36.063 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.061+0000 7f458c5df980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:36.072 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.070+0000 7f458c5df980 1 bluefs mount shared_bdev_used = 9568256 2023-08-16T02:06:36.072 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.070+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:36.075 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.072+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:36.075 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.072+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:36.075 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.072+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:36.075 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:36.075 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:36.076 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.073+0000 7f458c5df980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:36.462 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.460+0000 7f458c5df980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:36.464 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-08-16T02:06:36.464 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 0a 05 4c 02 00 00 67 72 0a da c5 4a 49 f5 9d ac |..L...gr...JI...| 2023-08-16T02:06:36.464 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 47 ee b7 a9 ba 6d 00 00 00 00 1d 00 00 00 01 00 |G....m..........| 2023-08-16T02:06:36.464 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:36.464 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-08-16T02:06:36.465 INFO:tasks.workunit.client.0.smithi150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-08-16T02:06:36.466 INFO:tasks.workunit.client.0.smithi150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-08-16T02:06:36.466 INFO:tasks.workunit.client.0.smithi150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-08-16T02:06:36.466 INFO:tasks.workunit.client.0.smithi150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-08-16T02:06:36.466 INFO:tasks.workunit.client.0.smithi150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-08-16T02:06:36.467 INFO:tasks.workunit.client.0.smithi150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-08-16T02:06:36.468 INFO:tasks.workunit.client.0.smithi150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000220 72 65 1d 00 00 00 05 00 00 00 d0 ae ff 71 f4 36 |re...........q.6| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000230 44 c5 83 02 df 02 60 22 ff 38 00 00 00 00 00 00 |D.....`".8......| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000240 00 00 1d 00 00 00 ce 2e dc 64 7e a2 fa 03 01 00 |.........d~.....| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000250 00 00 |..| 2023-08-16T02:06:36.469 INFO:tasks.workunit.client.0.smithi150.stdout:00000252 2023-08-16T02:06:36.470 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.470 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-08-16T02:06:36.470 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.470 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:36.470 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:36.471 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:36.471 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:36.471 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:36.472 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:0000003f 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:36.473 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:36.474 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:36.475 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:36.475 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:36.475 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.475 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:36.475 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:36.475 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:36.476 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:36.476 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:36.477 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:36.477 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.477 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:36.478 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:36.478 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:36.478 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:36.478 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:36.478 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:36.478 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.479 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:36.480 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:36.480 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:36.480 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:36.480 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:36.480 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:36.480 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.481 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-08-16T02:06:36.481 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.481 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-08-16T02:06:36.481 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:36.482 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.462+0000 7f458c5df980 1 bluefs umount 2023-08-16T02:06:36.482 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.462+0000 7f458c5df980 1 bdev(0x5560cb15e380 td/osd-mapper/0/block) close 2023-08-16T02:06:36.710 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.709+0000 7f458c5df980 1 freelist shutdown 2023-08-16T02:06:36.711 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:36.709+0000 7f458c5df980 1 bdev(0x5560cb15e000 td/osd-mapper/0/block) close 2023-08-16T02:06:36.802 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2023-08-16T02:06:36.802 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-08-16T02:06:36.803 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-08-16T02:06:37.851 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:37.851 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:37.851 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-08-16T02:06:37.852 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-08-16T02:06:37.852 INFO:tasks.workunit.client.0.smithi150.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:37.852 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-08-16T02:06:37.853 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.vyrhqxXxWF_the_val 2023-08-16T02:06:37.853 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:37.853 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.vyrhqxXxWF_the_val' 2023-08-16T02:06:37.854 INFO:tasks.workunit.client.0.smithi150.stdout:Value dumped in: /tmp/tmp.vyrhqxXxWF_the_val 2023-08-16T02:06:37.854 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.vyrhqxXxWF_the_val 2023-08-16T02:06:38.489 INFO:tasks.workunit.client.0.smithi150.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-08-16T02:06:38.900 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:38.901 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.vyrhqxXxWF_the_val 2023-08-16T02:06:38.908 INFO:tasks.workunit.client.0.smithi150.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-08-16T02:06:38.909 INFO:tasks.workunit.client.0.smithi150.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-08-16T02:06:38.909 INFO:tasks.workunit.client.0.smithi150.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-08-16T02:06:38.909 INFO:tasks.workunit.client.0.smithi150.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-08-16T02:06:38.909 INFO:tasks.workunit.client.0.smithi150.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-08-16T02:06:38.909 INFO:tasks.workunit.client.0.smithi150.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-08-16T02:06:38.909 INFO:tasks.workunit.client.0.smithi150.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-08-16T02:06:38.910 INFO:tasks.workunit.client.0.smithi150.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-08-16T02:06:38.910 INFO:tasks.workunit.client.0.smithi150.stdout:0000073 2023-08-16T02:06:38.910 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-08-16T02:06:38.911 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:39.954 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.vyrhqxXxWF_the_val 2023-08-16T02:06:41.000 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:41.000 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:06:41.001 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-08-16T02:06:41.001 INFO:tasks.workunit.client.0.smithi150.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2023-08-16T02:06:41.001 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2023-08-16T02:06:41.002 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-08-16T02:06:41.002 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-08-16T02:06:41.027 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.024+0000 7fd7dda87980 1 bdev(0x562d644f0000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-08-16T02:06:41.027 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.025+0000 7fd7dda87980 1 bdev(0x562d644f0000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:41.028 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.025+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-08-16T02:06:41.028 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.025+0000 7fd7dda87980 1 bdev(0x562d644f0380 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-08-16T02:06:41.028 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.025+0000 7fd7dda87980 1 bdev(0x562d644f0380 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:41.028 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.025+0000 7fd7dda87980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-08-16T02:06:41.028 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.025+0000 7fd7dda87980 1 bluefs mount 2023-08-16T02:06:41.030 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.029+0000 7fd7dda87980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:41.041 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.039+0000 7fd7dda87980 1 bluefs mount shared_bdev_used = 0 2023-08-16T02:06:41.041 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.039+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:41.043 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:41.043 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.041+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:41.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.042+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:41.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.042+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:41.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.042+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:41.048 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2023-08-16T02:06:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2023-08-16T02:06:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-08-16T02:06:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2023-08-16T02:06:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 freelist init 2023-08-16T02:06:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.047+0000 7fd7dda87980 1 freelist _read_cfg 2023-08-16T02:06:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.048+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbf000, fragmentation 6.10353e-07 2023-08-16T02:06:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.048+0000 7fd7dda87980 1 bluefs umount 2023-08-16T02:06:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.048+0000 7fd7dda87980 1 bdev(0x562d644f0380 td/osd-mapper/1/block) close 2023-08-16T02:06:41.291 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.290+0000 7fd7dda87980 1 bdev(0x562d644f0380 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2023-08-16T02:06:41.292 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.290+0000 7fd7dda87980 1 bdev(0x562d644f0380 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:41.292 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.290+0000 7fd7dda87980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2023-08-16T02:06:41.292 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.290+0000 7fd7dda87980 1 bluefs mount 2023-08-16T02:06:41.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.304+0000 7fd7dda87980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:41.315 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.313+0000 7fd7dda87980 1 bluefs mount shared_bdev_used = 9568256 2023-08-16T02:06:41.315 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.313+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:41.317 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:41.317 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:41.317 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:41.318 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:41.319 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.315+0000 7fd7dda87980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:41.578 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.577+0000 7fd7dda87980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:41.580 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-08-16T02:06:41.580 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 0a 05 4c 02 00 00 67 72 0a da c5 4a 49 f5 9d ac |..L...gr...JI...| 2023-08-16T02:06:41.580 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 47 ee b7 a9 ba 6d 01 00 00 00 1d 00 00 00 01 00 |G....m..........| 2023-08-16T02:06:41.580 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:41.580 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-08-16T02:06:41.581 INFO:tasks.workunit.client.0.smithi150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-08-16T02:06:41.582 INFO:tasks.workunit.client.0.smithi150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-08-16T02:06:41.583 INFO:tasks.workunit.client.0.smithi150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-08-16T02:06:41.584 INFO:tasks.workunit.client.0.smithi150.stdout:00000220 72 65 1d 00 00 00 0a 00 00 00 06 99 72 a8 6d ef |re..........r.m.| 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout:00000230 45 78 a1 46 b3 6c 62 99 a3 cc 00 00 00 00 00 00 |Ex.F.lb.........| 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout:00000240 00 00 1d 00 00 00 ef 2e dc 64 42 49 c8 08 01 00 |.........dBI....| 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout:00000250 00 00 |..| 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout:00000252 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.585 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.586 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:0000003f 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:41.587 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.588 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:41.589 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:41.590 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:41.591 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:41.592 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-08-16T02:06:41.593 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:41.594 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.577+0000 7fd7dda87980 1 bluefs umount 2023-08-16T02:06:41.594 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.577+0000 7fd7dda87980 1 bdev(0x562d644f0380 td/osd-mapper/1/block) close 2023-08-16T02:06:41.819 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.818+0000 7fd7dda87980 1 freelist shutdown 2023-08-16T02:06:41.819 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:41.818+0000 7fd7dda87980 1 bdev(0x562d644f0000 td/osd-mapper/1/block) close 2023-08-16T02:06:42.049 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-08-16T02:06:42.049 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-08-16T02:06:42.049 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-08-16T02:06:42.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:42.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:42.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-08-16T02:06:42.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-08-16T02:06:42.844 INFO:tasks.workunit.client.0.smithi150.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:42.844 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-08-16T02:06:42.845 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.0OkrznLD7h_the_val 2023-08-16T02:06:42.845 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:42.845 INFO:tasks.workunit.client.0.smithi150.stdout:Value dumped in: /tmp/tmp.0OkrznLD7h_the_val 2023-08-16T02:06:42.846 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.0OkrznLD7h_the_val' 2023-08-16T02:06:42.846 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.0OkrznLD7h_the_val 2023-08-16T02:06:43.389 INFO:tasks.workunit.client.0.smithi150.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-08-16T02:06:43.891 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:43.892 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.0OkrznLD7h_the_val 2023-08-16T02:06:43.892 INFO:tasks.workunit.client.0.smithi150.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-08-16T02:06:43.892 INFO:tasks.workunit.client.0.smithi150.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-08-16T02:06:43.892 INFO:tasks.workunit.client.0.smithi150.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-08-16T02:06:43.893 INFO:tasks.workunit.client.0.smithi150.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-08-16T02:06:43.893 INFO:tasks.workunit.client.0.smithi150.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-08-16T02:06:43.893 INFO:tasks.workunit.client.0.smithi150.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-08-16T02:06:43.893 INFO:tasks.workunit.client.0.smithi150.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-08-16T02:06:43.893 INFO:tasks.workunit.client.0.smithi150.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-08-16T02:06:43.893 INFO:tasks.workunit.client.0.smithi150.stdout:0000073 2023-08-16T02:06:43.894 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-08-16T02:06:43.894 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:44.941 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.0OkrznLD7h_the_val 2023-08-16T02:06:45.988 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:45.989 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:06:45.989 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2023-08-16T02:06:45.989 INFO:tasks.workunit.client.0.smithi150.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2023-08-16T02:06:45.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2023-08-16T02:06:45.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2023-08-16T02:06:45.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-08-16T02:06:46.016 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.013+0000 7fc933da1980 1 bdev(0x56249ad84000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-08-16T02:06:46.016 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.014+0000 7fc933da1980 1 bdev(0x56249ad84000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:46.016 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.014+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 data 0.06 2023-08-16T02:06:46.016 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.014+0000 7fc933da1980 1 bdev(0x56249ad84380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-08-16T02:06:46.017 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.014+0000 7fc933da1980 1 bdev(0x56249ad84380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:46.017 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.014+0000 7fc933da1980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-08-16T02:06:46.017 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.014+0000 7fc933da1980 1 bluefs mount 2023-08-16T02:06:46.027 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.025+0000 7fc933da1980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:46.036 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.035+0000 7fc933da1980 1 bluefs mount shared_bdev_used = 0 2023-08-16T02:06:46.037 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.035+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:46.039 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:46.039 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:46.040 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:46.040 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:46.040 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:46.040 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:46.040 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:46.040 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:46.041 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:46.041 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.037+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:46.041 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.038+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:46.041 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.038+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:46.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:46.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2023-08-16T02:06:46.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2023-08-16T02:06:46.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2023-08-16T02:06:46.045 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2023-08-16T02:06:46.046 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 freelist init 2023-08-16T02:06:46.046 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.043+0000 7fc933da1980 1 freelist _read_cfg 2023-08-16T02:06:46.047 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.044+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 12 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 4.19618e-07 2023-08-16T02:06:46.047 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.044+0000 7fc933da1980 1 bluefs umount 2023-08-16T02:06:46.048 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.044+0000 7fc933da1980 1 bdev(0x56249ad84380 td/osd-mapper/2/block) close 2023-08-16T02:06:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.283+0000 7fc933da1980 1 bdev(0x56249ad84380 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2023-08-16T02:06:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.283+0000 7fc933da1980 1 bdev(0x56249ad84380 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2023-08-16T02:06:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.283+0000 7fc933da1980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2023-08-16T02:06:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.283+0000 7fc933da1980 1 bluefs mount 2023-08-16T02:06:46.293 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.292+0000 7fc933da1980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2023-08-16T02:06:46.302 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.301+0000 7fc933da1980 1 bluefs mount shared_bdev_used = 9568256 2023-08-16T02:06:46.302 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.301+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2023-08-16T02:06:46.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2023-08-16T02:06:46.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2023-08-16T02:06:46.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2023-08-16T02:06:46.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2023-08-16T02:06:46.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.303+0000 7fc933da1980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2023-08-16T02:06:46.536 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.534+0000 7fc933da1980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2023-08-16T02:06:46.538 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2023-08-16T02:06:46.538 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 0a 05 4c 02 00 00 67 72 0a da c5 4a 49 f5 9d ac |..L...gr...JI...| 2023-08-16T02:06:46.538 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 47 ee b7 a9 ba 6d 02 00 00 00 1d 00 00 00 01 00 |G....m..........| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2023-08-16T02:06:46.539 INFO:tasks.workunit.client.0.smithi150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2023-08-16T02:06:46.540 INFO:tasks.workunit.client.0.smithi150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2023-08-16T02:06:46.541 INFO:tasks.workunit.client.0.smithi150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2023-08-16T02:06:46.542 INFO:tasks.workunit.client.0.smithi150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000220 72 65 1d 00 00 00 0f 00 00 00 bd 94 1c 8c 5b ea |re............[.| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000230 4e 6e 83 bb 99 26 81 b7 c4 0a 00 00 00 00 00 00 |Nn...&..........| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000240 00 00 1d 00 00 00 0f 2f dc 64 e4 12 03 38 01 00 |......./.d...8..| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000250 00 00 |..| 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout:00000252 2023-08-16T02:06:46.543 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:46.544 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:46.545 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:46.546 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2023-08-16T02:06:46.546 INFO:tasks.workunit.client.0.smithi150.stdout:0000003f 2023-08-16T02:06:46.546 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.546 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:46.546 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2023-08-16T02:06:46.546 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:00000047 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:46.547 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:46.548 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:46.549 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:46.549 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:46.549 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:46.549 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.549 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.550 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:06:46.551 INFO:tasks.workunit.client.0.smithi150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout:0000003b 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.552 INFO:tasks.workunit.client.0.smithi150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2023-08-16T02:06:46.553 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:06:46.553 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.536+0000 7fc933da1980 1 bluefs umount 2023-08-16T02:06:46.553 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.536+0000 7fc933da1980 1 bdev(0x56249ad84380 td/osd-mapper/2/block) close 2023-08-16T02:06:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.776+0000 7fc933da1980 1 freelist shutdown 2023-08-16T02:06:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:46.776+0000 7fc933da1980 1 bdev(0x56249ad84000 td/osd-mapper/2/block) close 2023-08-16T02:06:47.034 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2023-08-16T02:06:47.035 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2023-08-16T02:06:47.035 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-08-16T02:06:48.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:48.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:48.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2023-08-16T02:06:48.079 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2023-08-16T02:06:48.079 INFO:tasks.workunit.client.0.smithi150.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:48.079 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2023-08-16T02:06:48.080 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.oLuVTpzj1k_the_val 2023-08-16T02:06:48.080 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:48.081 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.oLuVTpzj1k_the_val' 2023-08-16T02:06:48.081 INFO:tasks.workunit.client.0.smithi150.stdout:Value dumped in: /tmp/tmp.oLuVTpzj1k_the_val 2023-08-16T02:06:48.081 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.oLuVTpzj1k_the_val 2023-08-16T02:06:48.547 INFO:tasks.workunit.client.0.smithi150.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2023-08-16T02:06:48.877 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:48.878 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.oLuVTpzj1k_the_val 2023-08-16T02:06:48.878 INFO:tasks.workunit.client.0.smithi150.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2023-08-16T02:06:48.878 INFO:tasks.workunit.client.0.smithi150.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout:0000060 0000 0100 0000 0000 0000 0000 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2023-08-16T02:06:48.879 INFO:tasks.workunit.client.0.smithi150.stdout:0000073 2023-08-16T02:06:48.880 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-08-16T02:06:48.880 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:06:49.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.oLuVTpzj1k_the_val 2023-08-16T02:06:50.975 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:06:50.976 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 ' 2023-08-16T02:06:50.976 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 2023-08-16T02:06:50.977 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:50.977 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:06:50.977 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/' 2023-08-16T02:06:50.977 INFO:tasks.workunit.client.0.smithi150.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ 2023-08-16T02:06:50.978 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2023-08-16T02:06:50.978 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2023-08-16T02:06:50.982 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:06:50.982 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:50.982 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2023-08-16T02:06:50.982 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2023-08-16T02:06:50.982 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-08-16T02:06:50.983 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2023-08-16T02:06:50.983 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-08-16T02:06:50.983 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2023-08-16T02:06:50.983 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:50.983 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:06:50.984 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:06:50.984 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:06:50.985 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2023-08-16T02:06:50.985 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2023-08-16T02:06:50.985 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-08-16T02:06:50.985 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-08-16T02:06:50.986 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-08-16T02:06:50.986 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-08-16T02:06:50.986 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:06:50.986 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:06:50.987 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:06:50.987 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:06:50.988 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:06:50.988 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93296/$cluster-$name.asok' 2023-08-16T02:06:50.988 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93296/$cluster-$name.asok' 2023-08-16T02:06:50.989 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:06:50.989 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-08-16T02:06:50.989 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-08-16T02:06:50.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:06:50.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:06:50.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:06:50.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:06:50.991 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-08-16T02:06:50.991 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-08-16T02:06:50.991 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2023-08-16T02:06:50.992 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2023-08-16T02:06:50.992 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.0 2023-08-16T02:06:50.994 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.93296/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-08-16T02:06:50.994 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2023-08-16T02:06:50.994 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2023-08-16T02:06:50.995 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-08-16T02:06:50.995 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-08-16T02:06:50.996 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-08-16T02:06:51.020 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:51.018+0000 7f2c854f3640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:51.023 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:51.020+0000 7f2c854f3640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:51.024 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:51.022+0000 7f2c854f3640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2023-08-16T02:06:51.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:06:51.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-08-16T02:06:51.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:06:51.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:06:51.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:51.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:06:51.314 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:06:51.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:51.315 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:06:51.551 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:51.548+0000 7f2c854f3640 -1 Falling back to public interface 2023-08-16T02:06:51.664 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:06:52.666 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:06:52.666 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:52.666 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:06:52.667 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:06:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:06:52.992 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:06:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:53.817+0000 7f2c854f3640 -1 osd.0 29 log_to_monitors true 2023-08-16T02:06:53.993 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:06:53.993 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:53.994 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:06:53.995 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:06:53.995 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:53.996 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:06:54.389 INFO:tasks.workunit.client.0.smithi150.stdout:osd.0 up in weight 1 up_from 32 up_thru 17 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6800/1724042547,v1:127.0.0.1:6801/1724042547] [v2:127.0.0.1:6802/1724042547,v1:127.0.0.1:6803/1724042547] exists,up d0aeff71-f436-44c5-8302-df026022ff38 2023-08-16T02:06:54.392 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:06:54.392 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:06:54.392 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:06:54.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:06:54.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:54.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2023-08-16T02:06:54.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2023-08-16T02:06:54.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-08-16T02:06:54.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2023-08-16T02:06:54.394 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-08-16T02:06:54.394 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2023-08-16T02:06:54.394 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:54.394 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:06:54.395 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:06:54.395 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:06:54.395 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2023-08-16T02:06:54.395 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2023-08-16T02:06:54.395 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-08-16T02:06:54.396 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-08-16T02:06:54.396 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-08-16T02:06:54.396 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-08-16T02:06:54.396 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:06:54.396 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:06:54.396 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93296/$cluster-$name.asok' 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93296/$cluster-$name.asok' 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-08-16T02:06:54.397 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-08-16T02:06:54.398 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2023-08-16T02:06:54.399 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2023-08-16T02:06:54.399 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.1 2023-08-16T02:06:54.400 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.93296/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-08-16T02:06:54.400 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2023-08-16T02:06:54.400 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2023-08-16T02:06:54.401 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-08-16T02:06:54.401 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-08-16T02:06:54.401 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-08-16T02:06:54.425 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:54.423+0000 7f69c5b3f640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:54.495+0000 7f69c5b3f640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:54.496+0000 7f69c5b3f640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:54.761 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2023-08-16T02:06:54.761 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:06:54.761 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-08-16T02:06:54.761 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:06:54.762 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:06:54.762 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:54.762 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:06:54.762 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:06:54.763 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:54.763 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:06:55.025 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:55.022+0000 7f69c5b3f640 -1 Falling back to public interface 2023-08-16T02:06:55.096 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:06:56.097 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:06:56.097 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:56.098 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:06:56.098 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:06:56.098 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:56.098 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:06:56.454 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:06:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:57.342+0000 7f69c5b3f640 -1 osd.1 29 log_to_monitors true 2023-08-16T02:06:57.455 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:06:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:57.456 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:06:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:06:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:06:57.931 INFO:tasks.workunit.client.0.smithi150.stdout:osd.1 up in weight 1 up_from 35 up_thru 17 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/3115010120,v1:127.0.0.1:6811/3115010120] [v2:127.0.0.1:6812/3115010120,v1:127.0.0.1:6813/3115010120] exists,up 069972a8-6def-4578-a146-b36c6299a3cc 2023-08-16T02:06:57.934 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:06:57.934 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:06:57.934 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2023-08-16T02:06:57.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2023-08-16T02:06:57.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:06:57.937 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93296/$cluster-$name.asok' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93296/$cluster-$name.asok' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:06:57.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:06:57.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:06:57.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:06:57.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2023-08-16T02:06:57.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2023-08-16T02:06:57.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2023-08-16T02:06:57.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2023-08-16T02:06:57.940 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.2 2023-08-16T02:06:57.940 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=67720ada-c54a-49f5-9dac-47eeb7a9ba6d --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.93296/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2023-08-16T02:06:57.941 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2023-08-16T02:06:57.941 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2023-08-16T02:06:57.941 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2023-08-16T02:06:57.941 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2023-08-16T02:06:57.941 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2023-08-16T02:06:57.969 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:57.966+0000 7f1f8b8e1640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:58.012 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:58.010+0000 7f1f8b8e1640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:58.013 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:58.010+0000 7f1f8b8e1640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:06:58.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2023-08-16T02:06:58.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:06:58.260 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-08-16T02:06:58.260 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:06:58.260 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:06:58.260 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:58.260 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:06:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:06:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:06:58.534 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:06:58.532+0000 7f1f8b8e1640 -1 Falling back to public interface 2023-08-16T02:06:58.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:06:59.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:06:59.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:06:59.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:06:59.753 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:06:59.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:06:59.754 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:07:00.082 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:00.833 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:00.830+0000 7f1f8b8e1640 -1 osd.2 29 log_to_monitors true 2023-08-16T02:07:01.084 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:01.084 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:01.084 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:07:01.085 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:07:01.085 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:01.085 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:07:01.490 INFO:tasks.workunit.client.0.smithi150.stdout:osd.2 up in weight 1 up_from 38 up_thru 17 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6818/1428151828,v1:127.0.0.1:6819/1428151828] [v2:127.0.0.1:6820/1428151828,v1:127.0.0.1:6821/1428151828] exists,up bd941c8c-5bea-4e6e-83bb-992681b7c40a 2023-08-16T02:07:01.491 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:07:01.491 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:07:01.491 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:07:01.491 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2023-08-16T02:07:02.493 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2023-08-16T02:07:02.495 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2023-08-16T02:07:02.496 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:07:02.496 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2023-08-16T02:07:02.626 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:07:02.626 INFO:tasks.workunit.client.0.smithi150.stdout: "version": "18.0.0-5526-gb5259484", 2023-08-16T02:07:02.626 INFO:tasks.workunit.client.0.smithi150.stdout: "release": "reef", 2023-08-16T02:07:02.626 INFO:tasks.workunit.client.0.smithi150.stdout: "release_type": "dev" 2023-08-16T02:07:02.627 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:07:02.637 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:07:02.637 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2023-08-16T02:07:02.756 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:07:02.757 INFO:tasks.workunit.client.0.smithi150.stdout: "version": "18.0.0-5526-gb5259484", 2023-08-16T02:07:02.757 INFO:tasks.workunit.client.0.smithi150.stdout: "release": "reef", 2023-08-16T02:07:02.757 INFO:tasks.workunit.client.0.smithi150.stdout: "release_type": "dev" 2023-08-16T02:07:02.757 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:07:02.766 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2023-08-16T02:07:02.766 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2023-08-16T02:07:02.886 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:07:02.887 INFO:tasks.workunit.client.0.smithi150.stdout: "version": "18.0.0-5526-gb5259484", 2023-08-16T02:07:02.888 INFO:tasks.workunit.client.0.smithi150.stdout: "release": "reef", 2023-08-16T02:07:02.888 INFO:tasks.workunit.client.0.smithi150.stdout: "release_type": "dev" 2023-08-16T02:07:02.888 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:07:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2023-08-16T02:07:02.925 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "objxxx", 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: "clones": [ 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [ 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 1, 2023-08-16T02:07:02.926 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap01" 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap02" 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 5, 2023-08-16T02:07:02.927 INFO:tasks.workunit.client.0.smithi150.stdout: "overlaps": [] 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [ 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2023-08-16T02:07:02.928 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap13" 2023-08-16T02:07:02.929 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:07:02.929 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2023-08-16T02:07:02.929 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 6, 2023-08-16T02:07:02.929 INFO:tasks.workunit.client.0.smithi150.stdout: "overlaps": [] 2023-08-16T02:07:02.929 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:07:02.929 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [ 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 4, 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap24" 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:07:02.930 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "snap25" 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 6, 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: "overlaps": [] 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2023-08-16T02:07:02.931 INFO:tasks.workunit.client.0.smithi150.stdout: { 2023-08-16T02:07:02.932 INFO:tasks.workunit.client.0.smithi150.stdout: "id": "head", 2023-08-16T02:07:02.932 INFO:tasks.workunit.client.0.smithi150.stdout: "snapshots": [], 2023-08-16T02:07:02.932 INFO:tasks.workunit.client.0.smithi150.stdout: "size": 6 2023-08-16T02:07:02.932 INFO:tasks.workunit.client.0.smithi150.stdout: } 2023-08-16T02:07:02.932 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2023-08-16T02:07:02.933 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:07:02.933 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2023-08-16T02:07:03.321 INFO:tasks.workunit.client.0.smithi150.stderr:nodeep-scrub is unset 2023-08-16T02:07:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2023-08-16T02:07:03.764 INFO:tasks.workunit.client.0.smithi150.stderr:noscrub is unset 2023-08-16T02:07:03.774 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2023-08-16T02:07:03.774 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2023-08-16T02:07:04.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2023-08-16T02:07:04.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2023-08-16T02:07:04.404 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:07:04.404 INFO:tasks.workunit.client.0.smithi150.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2023-08-16T02:07:02.324435+0000 27'7 40:62 [1,2,0] 1 [1,2,0] 1 27'7 2023-08-16T02:06:21.063288+0000 27'7 2023-08-16T02:06:21.063288+0000 0 1 periodic scrub scheduled @ 2023-08-17T02:06:21.063288+0000 4 0 2023-08-16T02:07:04.404 INFO:tasks.workunit.client.0.smithi150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2023-08-16T02:06:58.393558+0000 0'0 36:24 [0,1] 0 [0,1] 0 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:04.405 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:02.249211+0000 0'0 40:31 [1,0,2] 1 [1,0,2] 1 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:04.405 INFO:tasks.workunit.client.0.smithi150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2023-08-16T02:06:58.393564+0000 0'0 36:16 [0,1] 0 [0,1] 0 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:04.405 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:07:04.405 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:07:04.406 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:07:04.416 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2023-08-16T02:07:06.417 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-08-16T02:07:06.548 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:07:06.548 INFO:tasks.workunit.client.0.smithi150.stdout: "deep": true, 2023-08-16T02:07:06.549 INFO:tasks.workunit.client.0.smithi150.stdout: "stamp": "2023-08-09T02:05:26.545283+0000" 2023-08-16T02:07:06.549 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:07:06.557 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2023-08-16T02:07:11.558 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2023-08-16T02:07:11.862 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:07:11.862 INFO:tasks.workunit.client.0.smithi150.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2023-08-16T02:07:07.392157+0000 27'7 42:78 [1,2,0] 1 [1,2,0] 1 27'7 2023-08-16T02:07:07.392019+0000 27'7 2023-08-16T02:07:07.392019+0000 0 1 periodic scrub scheduled @ 2023-08-17T02:07:07.392019+0000 4 0 2023-08-16T02:07:11.862 INFO:tasks.workunit.client.0.smithi150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:01.821272+0000 0'0 42:39 [0,1,2] 0 [0,1,2] 0 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:11.862 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:02.249211+0000 0'0 42:34 [1,0,2] 1 [1,0,2] 1 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:11.862 INFO:tasks.workunit.client.0.smithi150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:01.720445+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:11.863 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:07:11.863 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:07:11.864 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:07:11.874 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:07:11.874 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-08-16T02:07:11.876 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:07:07.390+0000 7f69a02ae700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-08-16T02:07:11.876 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2023-08-16T02:07:11.878 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-08-16T02:07:11.878 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2023-08-16T02:07:11.879 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2023-08-16T02:07:11.880 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2023-08-16T02:07:11.880 INFO:tasks.workunit.client.0.smithi150.stdout:prev count: 1 2023-08-16T02:07:11.880 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep_scrub 2023-08-16T02:07:11.995 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:07:11.996 INFO:tasks.workunit.client.0.smithi150.stdout: "deep": true, 2023-08-16T02:07:11.996 INFO:tasks.workunit.client.0.smithi150.stdout: "stamp": "2023-08-09T02:05:31.994661+0000" 2023-08-16T02:07:11.996 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:07:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2023-08-16T02:07:17.007 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2023-08-16T02:07:17.308 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:07:17.309 INFO:tasks.workunit.client.0.smithi150.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2023-08-16T02:07:12.311602+0000 27'7 42:90 [1,2,0] 1 [1,2,0] 1 27'7 2023-08-16T02:07:12.311472+0000 27'7 2023-08-16T02:07:12.311472+0000 0 1 periodic scrub scheduled @ 2023-08-17T02:07:12.311472+0000 4 0 2023-08-16T02:07:17.309 INFO:tasks.workunit.client.0.smithi150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:01.821272+0000 0'0 42:39 [0,1,2] 0 [0,1,2] 0 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:17.309 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:02.249211+0000 0'0 42:34 [1,0,2] 1 [1,0,2] 1 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:17.309 INFO:tasks.workunit.client.0.smithi150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:07:01.720445+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2023-08-16T02:06:09.961256+0000 0'0 2023-08-16T02:06:09.961256+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:06:09.961256+0000 0 0 2023-08-16T02:07:17.309 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:07:17.309 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:07:17.310 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:07:17.320 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:07:17.320 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-08-16T02:07:17.322 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:07:07.390+0000 7f69a02ae700 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2023-08-16T02:07:17.322 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2023-08-16T02:07:17.323 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2023-08-16T02:07:17.324 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2023-08-16T02:07:17.324 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2023-08-16T02:07:17.325 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2023-08-16T02:07:17.325 INFO:tasks.workunit.client.0.smithi150.stdout:current count: 1 2023-08-16T02:07:17.326 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2023-08-16T02:07:17.326 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2023-08-16T02:07:17.326 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:07:17.326 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:07:17.326 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:07:17.326 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:07:17.327 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:07:17.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:07:17.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2023-08-16T02:07:17.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2023-08-16T02:07:17.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-08-16T02:07:17.633 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2023-08-16T02:07:17.633 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-08-16T02:07:19.176 INFO:tasks.workunit.client.0.smithi150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:07:19.177 INFO:tasks.workunit.client.0.smithi150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2023-08-16T02:07:19.177 INFO:tasks.workunit.client.0.smithi150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2023-08-16T02:07:19.177 INFO:tasks.workunit.client.0.smithi150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2023-08-16T02:07:19.177 INFO:tasks.workunit.client.0.smithi150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:07:19.177 INFO:tasks.workunit.client.0.smithi150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2023-08-16T02:07:19.178 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2023-08-16T02:07:19.178 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2023-08-16T02:07:19.178 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2023-08-16T02:07:19.178 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2023-08-16T02:07:19.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2023-08-16T02:07:19.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2023-08-16T02:07:19.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2023-08-16T02:07:19.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2023-08-16T02:07:19.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:07:19.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:07:20.081 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:07:20.081 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:07:20.082 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:07:20.082 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:07:20.083 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:07:20.083 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:07:20.084 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:07:20.085 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:07:20.085 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:07:20.086 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:07:20.086 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:07:20.086 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:07:20.088 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-08-16T02:07:20.088 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-08-16T02:07:20.103 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:07:20.103 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.103 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:07:20.104 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93296 2023-08-16T02:07:20.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:07:20.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:07:20.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2023-08-16T02:07:20.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-mapper 0 2023-08-16T02:07:20.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:07:20.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:07:20.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:07:20.107 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:07:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:07:20.108 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:07:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:07:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:07:20.110 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:07:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:07:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:07:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:07:20.112 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:07:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:07:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2023-08-16T02:07:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2023-08-16T02:07:20.115 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:07:20.115 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.115 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93296 2023-08-16T02:07:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93296 2023-08-16T02:07:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:07:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:07:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2023-08-16T02:07:20.117 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-16T02:07:20.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-16T02:07:20.199 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2023-08-16T02:07:20.200 DEBUG:teuthology.orchestra.run.smithi150:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5259484dbc61e8573f27e692560c6c107b9d4c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2023-08-16T02:07:20.263 INFO:tasks.workunit.client.0.smithi150.stderr:stty: 'standard input': Inappropriate ioctl for device 2023-08-16T02:07:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2023-08-16T02:07:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-08-16T02:07:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-08-16T02:07:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2023-08-16T02:07:20.271 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/osd-recovery-scrub 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2023-08-16T02:07:20.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2023-08-16T02:07:20.273 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2023-08-16T02:07:20.273 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2023-08-16T02:07:20.273 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2023-08-16T02:07:20.273 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2023-08-16T02:07:20.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none ' 2023-08-16T02:07:20.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2023-08-16T02:07:20.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2023-08-16T02:07:20.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr:TEST_recovery_scrub_2' 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2023-08-16T02:07:20.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.109770 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2023-08-16T02:07:20.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:07:20.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:07:20.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:07:20.280 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:07:20.281 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:07:20.281 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:07:20.281 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:07:20.281 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:07:20.282 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:07:20.282 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-08-16T02:07:20.282 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-08-16T02:07:20.282 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:07:20.282 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.282 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:20.283 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.109770 2023-08-16T02:07:20.283 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:07:20.283 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:07:20.283 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-08-16T02:07:20.284 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-08-16T02:07:20.284 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.284 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:20.285 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.109770 2023-08-16T02:07:20.286 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-08-16T02:07:20.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2023-08-16T02:07:20.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2023-08-16T02:07:20.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-08-16T02:07:20.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-08-16T02:07:20.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-08-16T02:07:20.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-08-16T02:07:20.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-08-16T02:07:20.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-08-16T02:07:20.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-08-16T02:07:20.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-08-16T02:07:20.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-08-16T02:07:20.685 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-08-16T02:07:20.686 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:07:20.686 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:07:20.686 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:07:20.686 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.686 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:20.686 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:20.687 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-08-16T02:07:20.853 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-08-16T02:07:20.854 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-08-16T02:07:20.854 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-08-16T02:07:20.854 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-08-16T02:07:20.854 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-08-16T02:07:20.855 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-08-16T02:07:20.855 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.109770/ceph-mon.a.asok 2023-08-16T02:07:20.856 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-08-16T02:07:20.857 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.109770/ceph-mon.a.asok config get fsid 2023-08-16T02:07:20.957 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-08-16T02:07:20.957 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-08-16T02:07:20.957 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-08-16T02:07:20.958 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-08-16T02:07:20.958 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-08-16T02:07:20.958 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-08-16T02:07:20.958 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-08-16T02:07:20.958 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-08-16T02:07:20.959 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-08-16T02:07:20.959 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:20.959 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:20.959 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.109770/ceph-mon.a.asok 2023-08-16T02:07:20.959 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-08-16T02:07:20.960 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.109770/ceph-mon.a.asok config get mon_host 2023-08-16T02:07:21.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2023-08-16T02:07:21.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-08-16T02:07:21.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-08-16T02:07:21.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-08-16T02:07:21.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-08-16T02:07:21.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-08-16T02:07:21.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-08-16T02:07:21.564 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-08-16T02:07:21.564 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:07:21.564 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:07:21.564 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:07:21.565 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:21.565 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:21.565 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:21.565 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-08-16T02:07:21.566 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-08-16T02:07:21.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2023-08-16T02:07:21.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2023-08-16T02:07:21.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2023-08-16T02:07:21.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2023-08-16T02:07:21.593 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2023-08-16T02:07:21.594 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2023-08-16T02:07:21.597 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:07:21.597 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2023-08-16T02:07:21.597 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:07:21.598 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:07:21.599 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:07:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:07:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:07:21.602 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:07:21.602 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-08-16T02:07:21.602 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-08-16T02:07:21.602 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:07:21.602 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=478763b0-b63a-4ebd-b5c2-b038227ea962 2023-08-16T02:07:21.602 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 478763b0-b63a-4ebd-b5c2-b038227ea962' 2023-08-16T02:07:21.603 INFO:tasks.workunit.client.0.smithi150.stdout:add osd0 478763b0-b63a-4ebd-b5c2-b038227ea962 2023-08-16T02:07:21.603 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:07:21.617 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZL9xkh5ilJBAACeef7n0LaiK/8s59xaY4Rw== 2023-08-16T02:07:21.617 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZL9xkh5ilJBAACeef7n0LaiK/8s59xaY4Rw=="}' 2023-08-16T02:07:21.617 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 478763b0-b63a-4ebd-b5c2-b038227ea962 -i td/osd-recovery-scrub/0/new.json 2023-08-16T02:07:21.903 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:07:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-08-16T02:07:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBZL9xkh5ilJBAACeef7n0LaiK/8s59xaY4Rw== --osd-uuid 478763b0-b63a-4ebd-b5c2-b038227ea962 2023-08-16T02:07:21.944 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:21.942+0000 7fa5f12e9640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:21.953 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:21.952+0000 7fa5f12e9640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:21.954 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:21.953+0000 7fa5f12e9640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:21.955 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:21.953+0000 7fa5f12e9640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-08-16T02:07:21.955 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:21.953+0000 7fa5f12e9640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-08-16T02:07:21.955 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:21.953+0000 7fa5f12e9640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-08-16T02:07:22.050 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:22.049+0000 7fa5f12e9640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-08-16T02:07:27.237 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-08-16T02:07:27.237 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:07:27.237 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-08-16T02:07:27.238 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd0 key to auth repository 2023-08-16T02:07:27.238 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:07:27.432 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-08-16T02:07:27.433 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.0 2023-08-16T02:07:27.433 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-08-16T02:07:27.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:07:27.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:07:27.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:07:27.460 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:27.459+0000 7f69954a6640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:27.462 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:27.461+0000 7f69954a6640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:27.463 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:27.462+0000 7f69954a6640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:27.645 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-08-16T02:07:27.645 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:07:27.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-08-16T02:07:27.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:07:27.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:07:27.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:27.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:07:27.646 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:07:27.647 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:27.647 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:07:27.854 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:28.001 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:28.000+0000 7f69954a6640 -1 Falling back to public interface 2023-08-16T02:07:28.855 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:28.856 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:28.856 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:07:28.857 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:07:28.857 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:28.857 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:07:29.178 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:30.179 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:30.180 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:30.180 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:07:30.180 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:07:30.181 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:30.181 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:07:30.320 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:30.318+0000 7f69954a6640 -1 osd.0 0 log_to_monitors true 2023-08-16T02:07:30.495 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:31.496 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:31.497 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:31.497 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:07:31.497 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:07:31.498 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:31.498 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:07:31.819 INFO:tasks.workunit.client.0.smithi150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2440990596,v1:127.0.0.1:6803/2440990596] [v2:127.0.0.1:6804/2440990596,v1:127.0.0.1:6805/2440990596] exists,up 478763b0-b63a-4ebd-b5c2-b038227ea962 2023-08-16T02:07:31.822 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:07:31.822 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:07:31.822 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:07:31.822 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:07:31.822 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:07:31.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:07:31.824 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:31.825 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:07:31.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:07:31.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:07:31.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-08-16T02:07:31.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-08-16T02:07:31.827 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:07:31.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cfac2c97-8cd4-4c4f-8882-38ce81baee3c 2023-08-16T02:07:31.828 INFO:tasks.workunit.client.0.smithi150.stdout:add osd1 cfac2c97-8cd4-4c4f-8882-38ce81baee3c 2023-08-16T02:07:31.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cfac2c97-8cd4-4c4f-8882-38ce81baee3c' 2023-08-16T02:07:31.828 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:07:31.841 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjL9xkh4YEMhAAZi675EB9h3+9sWP8Kos+HQ== 2023-08-16T02:07:31.841 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjL9xkh4YEMhAAZi675EB9h3+9sWP8Kos+HQ=="}' 2023-08-16T02:07:31.841 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cfac2c97-8cd4-4c4f-8882-38ce81baee3c -i td/osd-recovery-scrub/1/new.json 2023-08-16T02:07:32.223 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:07:32.235 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-08-16T02:07:32.236 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBjL9xkh4YEMhAAZi675EB9h3+9sWP8Kos+HQ== --osd-uuid cfac2c97-8cd4-4c4f-8882-38ce81baee3c 2023-08-16T02:07:32.265 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.263+0000 7f5567d94640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.302+0000 7f5567d94640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.303+0000 7f5567d94640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.304+0000 7f5567d94640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-08-16T02:07:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.304+0000 7f5567d94640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-08-16T02:07:32.306 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.304+0000 7f5567d94640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-08-16T02:07:32.374 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:32.372+0000 7f5567d94640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-08-16T02:07:37.857 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-08-16T02:07:37.858 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:07:37.859 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd1 key to auth repository 2023-08-16T02:07:37.859 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-08-16T02:07:37.860 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:07:38.263 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.1 2023-08-16T02:07:38.264 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-08-16T02:07:38.264 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-08-16T02:07:38.265 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:07:38.265 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:07:38.265 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:07:38.290 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:38.288+0000 7ff7f7c4a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:38.292 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:38.290+0000 7ff7f7c4a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:38.293 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:38.291+0000 7ff7f7c4a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:38.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-08-16T02:07:38.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:07:38.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-08-16T02:07:38.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:07:38.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:07:38.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:38.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:07:38.613 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:07:38.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:07:38.821 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:38.819+0000 7ff7f7c4a640 -1 Falling back to public interface 2023-08-16T02:07:38.957 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:39.958 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:39.958 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:39.958 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:07:39.959 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:07:39.959 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:39.959 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:07:40.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:41.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:41.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:41.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:07:41.279 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:07:41.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:41.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:07:41.284 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:41.282+0000 7ff7f7c4a640 -1 osd.1 0 log_to_monitors true 2023-08-16T02:07:41.717 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:42.718 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:42.718 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:42.718 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:07:42.719 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:07:42.719 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:42.719 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:07:43.039 INFO:tasks.workunit.client.0.smithi150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/653543687,v1:127.0.0.1:6811/653543687] [v2:127.0.0.1:6812/653543687,v1:127.0.0.1:6813/653543687] exists,up cfac2c97-8cd4-4c4f-8882-38ce81baee3c 2023-08-16T02:07:43.041 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:07:43.042 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-08-16T02:07:43.043 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:07:43.044 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:07:43.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-08-16T02:07:43.046 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-08-16T02:07:43.047 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:07:43.047 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f5b0c47-3898-4196-a600-88c772b50b5d 2023-08-16T02:07:43.047 INFO:tasks.workunit.client.0.smithi150.stdout:add osd2 7f5b0c47-3898-4196-a600-88c772b50b5d 2023-08-16T02:07:43.048 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7f5b0c47-3898-4196-a600-88c772b50b5d' 2023-08-16T02:07:43.048 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:07:43.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvL9xkgiWIAxAArK1NC1CiXphWSqX3BKYONw== 2023-08-16T02:07:43.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvL9xkgiWIAxAArK1NC1CiXphWSqX3BKYONw=="}' 2023-08-16T02:07:43.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f5b0c47-3898-4196-a600-88c772b50b5d -i td/osd-recovery-scrub/2/new.json 2023-08-16T02:07:43.426 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:07:43.438 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-08-16T02:07:43.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBvL9xkgiWIAxAArK1NC1CiXphWSqX3BKYONw== --osd-uuid 7f5b0c47-3898-4196-a600-88c772b50b5d 2023-08-16T02:07:43.468 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.466+0000 7f65517c3640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:43.502 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.499+0000 7f65517c3640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:43.503 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.500+0000 7f65517c3640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:43.503 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.501+0000 7f65517c3640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-08-16T02:07:43.503 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.501+0000 7f65517c3640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-08-16T02:07:43.504 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.501+0000 7f65517c3640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-08-16T02:07:43.579 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:43.576+0000 7f65517c3640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-08-16T02:07:49.035 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-08-16T02:07:49.035 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:07:49.036 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-08-16T02:07:49.036 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd2 key to auth repository 2023-08-16T02:07:49.036 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:07:49.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-08-16T02:07:49.438 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.2 2023-08-16T02:07:49.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:07:49.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-08-16T02:07:49.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:07:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:07:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:49.462+0000 7f9dd4ad0640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:49.464+0000 7f9dd4ad0640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:49.465+0000 7f9dd4ad0640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:49.782 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-08-16T02:07:49.782 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:07:49.782 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-08-16T02:07:49.783 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:07:49.783 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:07:49.783 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:49.783 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:07:49.783 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:07:49.784 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:49.784 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:07:49.986 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:49.983+0000 7f9dd4ad0640 -1 Falling back to public interface 2023-08-16T02:07:50.098 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:51.099 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:51.100 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:51.100 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:07:51.100 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:07:51.100 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:51.101 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:07:51.415 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:52.329 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:52.326+0000 7f9dd4ad0640 -1 osd.2 0 log_to_monitors true 2023-08-16T02:07:52.417 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:52.417 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:52.417 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:07:52.417 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:07:52.418 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:52.418 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:07:52.778 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:07:53.779 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:07:53.784 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:07:53.784 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:07:53.784 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:07:53.787 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:07:53.788 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:07:54.097 INFO:tasks.workunit.client.0.smithi150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/213634210,v1:127.0.0.1:6819/213634210] [v2:127.0.0.1:6820/213634210,v1:127.0.0.1:6821/213634210] exists,up 7f5b0c47-3898-4196-a600-88c772b50b5d 2023-08-16T02:07:54.102 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:07:54.103 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:07:54.103 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:07:54.103 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:07:54.103 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2023-08-16T02:07:54.104 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:07:54.104 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:07:54.104 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-08-16T02:07:54.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:07:54.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-08-16T02:07:54.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:07:54.105 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:07:54.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:07:54.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:07:54.106 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-08-16T02:07:54.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-08-16T02:07:54.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:07:54.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:07:54.108 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:07:54.108 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:07:54.108 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:07:54.109 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:07:54.109 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:07:54.109 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:07:54.109 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:07:54.110 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:54.110 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:07:54.110 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:07:54.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:07:54.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:07:54.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:07:54.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:07:54.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:07:54.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:07:54.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:07:54.112 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:07:54.113 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:07:54.113 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2023-08-16T02:07:54.113 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-08-16T02:07:54.113 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:07:54.114 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=335af41b-c907-47df-8a20-aa2e90db9e49 2023-08-16T02:07:54.114 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 335af41b-c907-47df-8a20-aa2e90db9e49' 2023-08-16T02:07:54.115 INFO:tasks.workunit.client.0.smithi150.stdout:add osd3 335af41b-c907-47df-8a20-aa2e90db9e49 2023-08-16T02:07:54.115 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:07:54.120 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6L9xkq6oEBxAA2Dt7g/GR7CDJmAipRy9DsA== 2023-08-16T02:07:54.120 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6L9xkq6oEBxAA2Dt7g/GR7CDJmAipRy9DsA=="}' 2023-08-16T02:07:54.120 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 335af41b-c907-47df-8a20-aa2e90db9e49 -i td/osd-recovery-scrub/3/new.json 2023-08-16T02:07:54.495 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:07:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-08-16T02:07:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB6L9xkq6oEBxAA2Dt7g/GR7CDJmAipRy9DsA== --osd-uuid 335af41b-c907-47df-8a20-aa2e90db9e49 2023-08-16T02:07:54.537 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.534+0000 7fc48cfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:54.580 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.577+0000 7fc48cfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:54.580 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.578+0000 7fc48cfd9640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:07:54.581 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.578+0000 7fc48cfd9640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-08-16T02:07:54.581 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.578+0000 7fc48cfd9640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-08-16T02:07:54.582 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.578+0000 7fc48cfd9640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-08-16T02:07:54.643 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:07:54.640+0000 7fc48cfd9640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-08-16T02:08:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-08-16T02:08:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:08:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-08-16T02:08:00.406 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd3 key to auth repository 2023-08-16T02:08:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:08:00.815 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.3 2023-08-16T02:08:00.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-08-16T02:08:00.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2023-08-16T02:08:00.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:08:00.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:08:00.817 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:08:00.842 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:08:00.839+0000 7f304a7cc640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:08:00.845 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:08:00.842+0000 7f304a7cc640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:08:00.846 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:08:00.843+0000 7f304a7cc640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:08:01.150 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-08-16T02:08:01.150 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:08:01.150 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-08-16T02:08:01.150 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:08:01.150 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:08:01.150 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:08:01.151 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:08:01.151 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:08:01.151 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:08:01.152 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:08:01.394 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:08:01.391+0000 7f304a7cc640 -1 Falling back to public interface 2023-08-16T02:08:01.461 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:08:02.462 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:08:02.462 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:08:02.463 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:08:02.463 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:08:02.463 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:08:02.463 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:08:02.785 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:08:03.775 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:08:03.773+0000 7f304a7cc640 -1 osd.3 0 log_to_monitors true 2023-08-16T02:08:03.786 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:08:03.787 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:08:03.787 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:08:03.788 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:08:03.788 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:08:03.788 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:08:04.164 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:08:05.165 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:08:05.166 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:08:05.166 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:08:05.167 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:08:05.167 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:08:05.167 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:08:05.526 INFO:tasks.workunit.client.0.smithi150.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2283796383,v1:127.0.0.1:6827/2283796383] [v2:127.0.0.1:6828/2283796383,v1:127.0.0.1:6829/2283796383] exists,up 335af41b-c907-47df-8a20-aa2e90db9e49 2023-08-16T02:08:05.527 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:08:05.527 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:08:05.527 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:08:05.528 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2023-08-16T02:08:05.528 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2023-08-16T02:08:05.996 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'test' created 2023-08-16T02:08:06.005 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-08-16T02:08:07.007 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2023-08-16T02:08:07.007 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-08-16T02:08:07.007 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-08-16T02:08:07.007 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-08-16T02:08:07.008 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-08-16T02:08:07.008 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-08-16T02:08:07.008 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-08-16T02:08:07.009 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-08-16T02:08:07.009 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-08-16T02:08:07.009 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-08-16T02:08:07.009 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-08-16T02:08:07.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-08-16T02:08:07.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-08-16T02:08:07.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-08-16T02:08:07.107 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-08-16T02:08:07.108 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-08-16T02:08:07.421 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-08-16T02:08:07.421 INFO:tasks.workunit.client.0.smithi150.stderr:1 2023-08-16T02:08:07.422 INFO:tasks.workunit.client.0.smithi150.stderr:2 2023-08-16T02:08:07.422 INFO:tasks.workunit.client.0.smithi150.stderr:3' 2023-08-16T02:08:07.422 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-08-16T02:08:07.422 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:08:07.422 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-08-16T02:08:07.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836489 2023-08-16T02:08:07.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836489 2023-08-16T02:08:07.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836489' 2023-08-16T02:08:07.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:08:07.566 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-08-16T02:08:07.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672966 2023-08-16T02:08:07.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672966 2023-08-16T02:08:07.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2023-08-16T02:08:07.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:08:07.692 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-08-16T02:08:07.818 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509444 2023-08-16T02:08:07.818 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509444 2023-08-16T02:08:07.819 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2023-08-16T02:08:07.819 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:08:07.819 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-08-16T02:08:07.943 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345922 2023-08-16T02:08:07.944 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345922 2023-08-16T02:08:07.944 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444 3-85899345922' 2023-08-16T02:08:07.944 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:08:07.945 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836489 2023-08-16T02:08:07.945 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:08:07.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-08-16T02:08:07.945 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836489 2023-08-16T02:08:07.946 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:08:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836489 2023-08-16T02:08:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2023-08-16T02:08:07.947 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.0 seq 21474836489 2023-08-16T02:08:07.947 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-08-16T02:08:08.253 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836489 -lt 21474836489 2023-08-16T02:08:08.253 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:08:08.253 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672966 2023-08-16T02:08:08.254 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:08:08.254 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-08-16T02:08:08.255 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672966 2023-08-16T02:08:08.255 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:08:08.255 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672966 2023-08-16T02:08:08.256 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.1 seq 42949672966 2023-08-16T02:08:08.256 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2023-08-16T02:08:08.256 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-08-16T02:08:08.607 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672965 -lt 42949672966 2023-08-16T02:08:08.608 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-08-16T02:08:09.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-08-16T02:08:09.609 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-08-16T02:08:09.934 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672967 -lt 42949672966 2023-08-16T02:08:09.935 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:08:09.936 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509444 2023-08-16T02:08:09.936 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:08:09.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-08-16T02:08:09.938 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509444 2023-08-16T02:08:09.938 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:08:09.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509444 2023-08-16T02:08:09.939 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2023-08-16T02:08:09.939 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.2 seq 64424509444 2023-08-16T02:08:09.939 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-08-16T02:08:10.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509445 -lt 64424509444 2023-08-16T02:08:10.281 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:08:10.281 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345922 2023-08-16T02:08:10.281 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:08:10.282 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-08-16T02:08:10.282 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345922 2023-08-16T02:08:10.283 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:08:10.283 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345922 2023-08-16T02:08:10.284 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2023-08-16T02:08:10.284 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.3 seq 85899345922 2023-08-16T02:08:10.284 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-08-16T02:08:10.594 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345922 -lt 85899345922 2023-08-16T02:08:10.595 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-08-16T02:08:10.595 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-08-16T02:08:10.595 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-08-16T02:08:11.026 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2023-08-16T02:08:11.026 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-08-16T02:08:11.027 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-08-16T02:08:11.027 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-08-16T02:08:11.027 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-08-16T02:08:11.027 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-08-16T02:08:11.027 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-08-16T02:08:11.027 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-08-16T02:08:11.376 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2023-08-16T02:08:11.377 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-08-16T02:08:11.377 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-08-16T02:08:11.377 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-08-16T02:08:11.786 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2023-08-16T02:08:11.787 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-08-16T02:08:11.787 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-08-16T02:08:11.787 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2023-08-16T02:08:11.788 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2023-08-16T02:08:11.788 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2023-08-16T02:08:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2023-08-16T02:08:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2023-08-16T02:08:12.393 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:08:12.394 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:08:06.704493+0000 0'0 23:8 [1] 1 [1] 1 0'0 2023-08-16T02:08:05.883116+0000 0'0 2023-08-16T02:08:05.883116+0000 0 0 periodic scrub scheduled @ 2023-08-17T08:23:43.456054+0000 0 0 2023-08-16T02:08:12.394 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:08:12.394 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:08:12.396 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:08:12.405 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.109770 bs=1M count=50 2023-08-16T02:08:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:50+0 records in 2023-08-16T02:08:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:50+0 records out 2023-08-16T02:08:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.226901 s, 231 MB/s 2023-08-16T02:08:12.634 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2023-08-16T02:08:12.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:12.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.109770 2023-08-16T02:08:14.200 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:14.200 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.109770 2023-08-16T02:08:15.606 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:15.606 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.109770 2023-08-16T02:08:17.154 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:17.155 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.109770 2023-08-16T02:08:18.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:18.619 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.109770 2023-08-16T02:08:20.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:20.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.109770 2023-08-16T02:08:21.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:21.824 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.109770 2023-08-16T02:08:23.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:23.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.109770 2023-08-16T02:08:24.928 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:24.928 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.109770 2023-08-16T02:08:26.333 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.109770 2023-08-16T02:08:27.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:27.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.109770 2023-08-16T02:08:29.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:29.334 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.109770 2023-08-16T02:08:30.869 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:30.869 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.109770 2023-08-16T02:08:32.375 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:32.375 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.109770 2023-08-16T02:08:33.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:33.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.109770 2023-08-16T02:08:35.337 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:35.337 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.109770 2023-08-16T02:08:36.902 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:36.902 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.109770 2023-08-16T02:08:38.458 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:38.458 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.109770 2023-08-16T02:08:39.897 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:39.897 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.109770 2023-08-16T02:08:41.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:41.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.109770 2023-08-16T02:08:42.995 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:42.996 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.109770 2023-08-16T02:08:44.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:44.393 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.109770 2023-08-16T02:08:45.857 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:45.858 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.109770 2023-08-16T02:08:47.380 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:47.380 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.109770 2023-08-16T02:08:49.186 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:49.187 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.109770 2023-08-16T02:08:50.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:50.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.109770 2023-08-16T02:08:52.099 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:52.100 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.109770 2023-08-16T02:08:53.739 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:53.739 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.109770 2023-08-16T02:08:55.329 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:55.329 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.109770 2023-08-16T02:08:57.010 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:57.011 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.109770 2023-08-16T02:08:58.516 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:58.516 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.109770 2023-08-16T02:08:59.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:08:59.965 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.109770 2023-08-16T02:09:01.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:01.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.109770 2023-08-16T02:09:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.109770 2023-08-16T02:09:04.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:04.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.109770 2023-08-16T02:09:05.906 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:05.906 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.109770 2023-08-16T02:09:07.420 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:07.421 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.109770 2023-08-16T02:09:09.019 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:09.019 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.109770 2023-08-16T02:09:10.457 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:10.458 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.109770 2023-08-16T02:09:11.997 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:11.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.109770 2023-08-16T02:09:13.512 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:13.513 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.109770 2023-08-16T02:09:14.986 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:14.986 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.109770 2023-08-16T02:09:16.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:16.407 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.109770 2023-08-16T02:09:17.922 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:17.923 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.109770 2023-08-16T02:09:19.396 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:19.396 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.109770 2023-08-16T02:09:20.944 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:20.944 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.109770 2023-08-16T02:09:22.399 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:22.400 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.109770 2023-08-16T02:09:23.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:23.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.109770 2023-08-16T02:09:25.487 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:25.488 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.109770 2023-08-16T02:09:26.876 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:26.877 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.109770 2023-08-16T02:09:28.324 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:28.324 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.109770 2023-08-16T02:09:29.788 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:29.789 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.109770 2023-08-16T02:09:31.237 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:31.237 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.109770 2023-08-16T02:09:32.676 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:32.676 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.109770 2023-08-16T02:09:34.140 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:34.141 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.109770 2023-08-16T02:09:35.588 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:35.589 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.109770 2023-08-16T02:09:37.128 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:37.128 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.109770 2023-08-16T02:09:38.642 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:38.643 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.109770 2023-08-16T02:09:40.207 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:40.208 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.109770 2023-08-16T02:09:41.722 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:41.722 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.109770 2023-08-16T02:09:43.247 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:43.248 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.109770 2023-08-16T02:09:44.678 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:44.678 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.109770 2023-08-16T02:09:46.184 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:46.184 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.109770 2023-08-16T02:09:47.682 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:47.683 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.109770 2023-08-16T02:09:49.205 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:49.205 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.109770 2023-08-16T02:09:50.552 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:50.553 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.109770 2023-08-16T02:09:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:52.134 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.109770 2023-08-16T02:09:53.615 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:53.615 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.109770 2023-08-16T02:09:55.087 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:55.088 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.109770 2023-08-16T02:09:56.493 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:56.493 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.109770 2023-08-16T02:09:58.041 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:58.041 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.109770 2023-08-16T02:09:59.556 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:09:59.556 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.109770 2023-08-16T02:10:01.129 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:01.129 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.109770 2023-08-16T02:10:02.502 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:02.502 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.109770 2023-08-16T02:10:04.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:04.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.109770 2023-08-16T02:10:05.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:05.823 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.109770 2023-08-16T02:10:07.412 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:07.412 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.109770 2023-08-16T02:10:09.052 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:09.052 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.109770 2023-08-16T02:10:10.760 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:10.760 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.109770 2023-08-16T02:10:12.297 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:12.298 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.109770 2023-08-16T02:10:13.979 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:13.979 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.109770 2023-08-16T02:10:15.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:15.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.109770 2023-08-16T02:10:17.195 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:17.195 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.109770 2023-08-16T02:10:18.664 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:18.664 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.109770 2023-08-16T02:10:20.204 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:20.204 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.109770 2023-08-16T02:10:21.643 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:21.643 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.109770 2023-08-16T02:10:23.208 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:23.208 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.109770 2023-08-16T02:10:24.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:24.622 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.109770 2023-08-16T02:10:26.195 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:26.195 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.109770 2023-08-16T02:10:27.676 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:27.676 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.109770 2023-08-16T02:10:29.191 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:29.191 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.109770 2023-08-16T02:10:30.757 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:30.758 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.109770 2023-08-16T02:10:32.294 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:32.295 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.109770 2023-08-16T02:10:33.842 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.109770 2023-08-16T02:10:35.332 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:35.332 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.109770 2023-08-16T02:10:36.896 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:36.897 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.109770 2023-08-16T02:10:38.386 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:38.386 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.109770 2023-08-16T02:10:39.950 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:39.951 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.109770 2023-08-16T02:10:41.606 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:41.607 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.109770 2023-08-16T02:10:43.163 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2023-08-16T02:10:43.163 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.109770 2023-08-16T02:10:44.585 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.109770 2023-08-16T02:10:44.591 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2023-08-16T02:10:45.019 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 1 size to 4 2023-08-16T02:10:45.029 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2023-08-16T02:10:45.030 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2023-08-16T02:10:45.030 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-08-16T02:10:45.030 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-08-16T02:10:45.030 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-08-16T02:10:45.031 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-08-16T02:10:45.329 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-08-16T02:10:47.340 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2023-08-16T02:10:47.341 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2023-08-16T02:10:47.342 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2023-08-16T02:10:47.342 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-08-16T02:10:47.343 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-08-16T02:10:47.343 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-08-16T02:10:47.343 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-08-16T02:10:47.647 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:10:47.662 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-08-16T02:10:49.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2023-08-16T02:10:49.659 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2023-08-16T02:10:49.660 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2023-08-16T02:10:49.660 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-08-16T02:10:49.661 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-08-16T02:10:49.661 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-08-16T02:10:49.661 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-08-16T02:10:49.980 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:10:49.990 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-08-16T02:10:51.991 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2023-08-16T02:10:51.992 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2023-08-16T02:10:51.992 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2023-08-16T02:10:51.993 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-08-16T02:10:51.993 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-08-16T02:10:51.993 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-08-16T02:10:51.994 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-08-16T02:10:52.303 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:10:52.313 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2023-08-16T02:10:54.314 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2023-08-16T02:10:54.315 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2023-08-16T02:10:54.315 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2023-08-16T02:10:54.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2023-08-16T02:10:54.316 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2023-08-16T02:10:54.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2023-08-16T02:10:54.317 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2023-08-16T02:10:54.743 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:10:54.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2023-08-16T02:10:54.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2023-08-16T02:10:54.754 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2023-08-16T02:10:55.188 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:10:55.189 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2023-08-16T02:10:49.663756+0000 24'1300 27:1322 [1,0,2,3] 1 [1,0] 1 0'0 2023-08-16T02:08:05.883116+0000 0'0 2023-08-16T02:08:05.883116+0000 0 0 periodic scrub scheduled @ 2023-08-17T04:49:57.362189+0000 0 0 2023-08-16T02:10:55.189 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:10:55.189 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:10:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:10:55.201 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2023-08-16T02:11:05.202 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2023-08-16T02:11:05.203 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:11:05.203 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:11:05.203 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:11:05.204 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:11:05.204 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:11:10.518 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:11:10.519 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2023-08-16T02:11:10.519 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='not scheduling scrubs due to active recovery' 2023-08-16T02:11:10.519 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2023-08-16T02:11:10.520 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2023-08-16T02:11:10.521 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.521 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2023-08-16T02:11:10.523 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:50.389+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.524 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:52.383+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.524 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:58.336+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.524 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:59.334+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.524 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:02.384+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.524 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:04.381+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.524 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:06.381+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.525 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:07.429+0000 7f6986c43700 15 osd.0 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.525 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.525 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2023-08-16T02:11:10.537 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:51.970+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.538 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:52.990+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.538 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:53.960+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.538 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:54.928+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.538 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:55.918+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.538 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:56.893+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.539 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:01.903+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.539 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:02.865+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.539 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:03.910+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.539 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:06.840+0000 7ff7e93e7700 15 osd.1 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.540 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.540 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2023-08-16T02:11:10.541 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:55.458+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.541 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:56.496+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.541 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:58.501+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.541 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:59.495+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.541 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:00.515+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.541 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:02.492+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.542 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:03.526+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.542 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:11:06.557+0000 7f9dc626d700 15 osd.2 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.542 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.542 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2023-08-16T02:11:10.543 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:51.817+0000 7f303bf69700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.543 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:54.774+0000 7f303bf69700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.544 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:57.800+0000 7f303bf69700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.544 INFO:tasks.workunit.client.0.smithi150.stdout:2023-08-16T02:10:59.713+0000 7f303bf69700 15 osd.3 27 sched_scrub: not scheduling scrubs due to active recovery 2023-08-16T02:11:10.544 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2023-08-16T02:11:10.545 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2023-08-16T02:11:10.545 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2023-08-16T02:11:10.545 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2023-08-16T02:11:10.545 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2023-08-16T02:11:10.546 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.546 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2023-08-16T02:11:10.548 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-08-16T02:11:10.548 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2023-08-16T02:11:10.549 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2023-08-16T02:11:10.549 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.549 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2023-08-16T02:11:10.560 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-08-16T02:11:10.560 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2023-08-16T02:11:10.562 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2023-08-16T02:11:10.562 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.562 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2023-08-16T02:11:10.563 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-08-16T02:11:10.563 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2023-08-16T02:11:10.564 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2023-08-16T02:11:10.564 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:10.564 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2023-08-16T02:11:10.566 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2023-08-16T02:11:10.566 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2023-08-16T02:11:10.568 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2023-08-16T02:11:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2023-08-16T02:11:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2023-08-16T02:11:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2023-08-16T02:11:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-08-16T02:11:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-08-16T02:11:10.570 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-08-16T02:11:10.570 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:11:10.570 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:11:10.570 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:11:10.571 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:11:10.571 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:11:10.574 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:11:10.574 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:11:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:11:10.577 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:11:10.577 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:11:10.577 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:11:10.577 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:11:10.578 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:11:10.578 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:11:10.579 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:11:10.579 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:11:10.579 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:11:10.580 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-08-16T02:11:10.581 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-08-16T02:11:10.601 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:11:10.602 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:10.602 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:10.602 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.109770 2023-08-16T02:11:10.603 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:11:10.603 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:11:10.603 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2023-08-16T02:11:10.603 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2023-08-16T02:11:10.604 INFO:tasks.workunit.client.0.smithi150.stdout:TEST PASSED 2023-08-16T02:11:10.608 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2023-08-16T02:11:10.608 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2023-08-16T02:11:10.608 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2023-08-16T02:11:10.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2023-08-16T02:11:10.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2023-08-16T02:11:10.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.109770 2023-08-16T02:11:10.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2023-08-16T02:11:10.610 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2023-08-16T02:11:10.610 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2023-08-16T02:11:10.610 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2023-08-16T02:11:10.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2023-08-16T02:11:10.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2023-08-16T02:11:10.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-08-16T02:11:10.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2023-08-16T02:11:10.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-08-16T02:11:10.612 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:11:10.612 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:11:10.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:11:10.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:11:10.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:11:10.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:11:10.614 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:11:10.614 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:11:10.615 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:11:10.615 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:11:10.615 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:11:10.616 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:11:10.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:11:10.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:10.619 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.109770 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:10.620 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.109770 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2023-08-16T02:11:10.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2023-08-16T02:11:10.622 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2023-08-16T02:11:10.622 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2023-08-16T02:11:10.622 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2023-08-16T02:11:10.622 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-08-16T02:11:10.957 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2023-08-16T02:11:10.958 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:10.958 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:10.958 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:10.958 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:10.958 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:10.958 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:10.960 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2023-08-16T02:11:11.142 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2023-08-16T02:11:11.143 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2023-08-16T02:11:11.143 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-08-16T02:11:11.143 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-08-16T02:11:11.144 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2023-08-16T02:11:11.144 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2023-08-16T02:11:11.144 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-08-16T02:11:11.144 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-08-16T02:11:11.144 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-08-16T02:11:11.146 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-08-16T02:11:11.146 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:11.146 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:11.147 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.109770/ceph-mon.a.asok 2023-08-16T02:11:11.147 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-08-16T02:11:11.147 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.109770/ceph-mon.a.asok config get fsid 2023-08-16T02:11:11.247 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2023-08-16T02:11:11.247 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2023-08-16T02:11:11.247 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2023-08-16T02:11:11.247 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2023-08-16T02:11:11.247 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:11.248 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.109770/ceph-mon.a.asok 2023-08-16T02:11:11.249 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2023-08-16T02:11:11.249 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.109770/ceph-mon.a.asok config get mon_host 2023-08-16T02:11:11.349 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2023-08-16T02:11:11.349 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2023-08-16T02:11:11.349 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2023-08-16T02:11:11.350 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2023-08-16T02:11:11.350 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2023-08-16T02:11:11.350 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2023-08-16T02:11:11.350 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2023-08-16T02:11:12.237 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2023-08-16T02:11:12.238 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:12.238 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:12.238 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:12.238 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:12.238 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:12.239 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:12.239 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-08-16T02:11:12.239 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2023-08-16T02:11:12.268 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2023-08-16T02:11:12.269 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2023-08-16T02:11:12.269 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2023-08-16T02:11:12.269 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2023-08-16T02:11:12.269 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2023-08-16T02:11:12.274 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2023-08-16T02:11:12.274 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2023-08-16T02:11:12.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:11:12.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:12.277 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:12.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:12.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:11:12.278 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:11:12.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:11:12.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:11:12.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:11:12.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:11:12.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:11:12.279 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a140c63-cecf-4a55-ae02-aed6d648572a 2023-08-16T02:11:12.280 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3a140c63-cecf-4a55-ae02-aed6d648572a' 2023-08-16T02:11:12.281 INFO:tasks.workunit.client.0.smithi150.stdout:add osd0 3a140c63-cecf-4a55-ae02-aed6d648572a 2023-08-16T02:11:12.281 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:11:12.292 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAMNxkmENHERAAhTC/LAbsFwuoUoqgUGe2AQ== 2023-08-16T02:11:12.292 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAMNxkmENHERAAhTC/LAbsFwuoUoqgUGe2AQ=="}' 2023-08-16T02:11:12.292 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a140c63-cecf-4a55-ae02-aed6d648572a -i td/osd-recovery-scrub/0/new.json 2023-08-16T02:11:12.575 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:11:12.586 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2023-08-16T02:11:12.587 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBAMNxkmENHERAAhTC/LAbsFwuoUoqgUGe2AQ== --osd-uuid 3a140c63-cecf-4a55-ae02-aed6d648572a 2023-08-16T02:11:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.617+0000 7f6485dae640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.623+0000 7f6485dae640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.624+0000 7f6485dae640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.625+0000 7f6485dae640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-08-16T02:11:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.625+0000 7f6485dae640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-08-16T02:11:12.641 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.625+0000 7f6485dae640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2023-08-16T02:11:12.730 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:12.727+0000 7f6485dae640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2023-08-16T02:11:17.946 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2023-08-16T02:11:17.946 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:11:17.947 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2023-08-16T02:11:17.947 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd0 key to auth repository 2023-08-16T02:11:17.947 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:11:18.140 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2023-08-16T02:11:18.141 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.0 2023-08-16T02:11:18.141 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:11:18.141 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:18.141 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:11:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:11:18.171 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:18.169+0000 7f90c1834640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:18.174 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:18.171+0000 7f90c1834640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:18.175 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:18.172+0000 7f90c1834640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:18.433 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2023-08-16T02:11:18.433 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:11:18.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2023-08-16T02:11:18.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:11:18.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:11:18.434 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:18.434 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:11:18.436 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:11:18.436 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:18.436 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:11:18.707 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:18.705+0000 7f90c1834640 -1 Falling back to public interface 2023-08-16T02:11:18.768 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:19.770 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:19.770 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:19.770 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:11:19.770 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:11:19.771 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:19.771 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:11:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:21.090 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:21.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:21.091 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:11:21.092 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:11:21.092 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:21.092 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:11:21.093 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:21.090+0000 7f90c1834640 -1 osd.0 0 log_to_monitors true 2023-08-16T02:11:21.469 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:22.470 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:22.470 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:22.470 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:11:22.471 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:11:22.471 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:22.471 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2023-08-16T02:11:22.822 INFO:tasks.workunit.client.0.smithi150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/262618216,v1:127.0.0.1:6803/262618216] [v2:127.0.0.1:6804/262618216,v1:127.0.0.1:6805/262618216] exists,up 3a140c63-cecf-4a55-ae02-aed6d648572a 2023-08-16T02:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:11:22.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2023-08-16T02:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2023-08-16T02:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2023-08-16T02:11:22.834 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:11:22.834 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf76eb5b-4443-4fef-9329-f9115f67fcec 2023-08-16T02:11:22.834 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bf76eb5b-4443-4fef-9329-f9115f67fcec' 2023-08-16T02:11:22.835 INFO:tasks.workunit.client.0.smithi150.stdout:add osd1 bf76eb5b-4443-4fef-9329-f9115f67fcec 2023-08-16T02:11:22.835 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKMNxk1ccyMhAA1zLmdSH0lM8wetmqR3s1/g== 2023-08-16T02:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKMNxk1ccyMhAA1zLmdSH0lM8wetmqR3s1/g=="}' 2023-08-16T02:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf76eb5b-4443-4fef-9329-f9115f67fcec -i td/osd-recovery-scrub/1/new.json 2023-08-16T02:11:23.225 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:11:23.237 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2023-08-16T02:11:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBKMNxk1ccyMhAA1zLmdSH0lM8wetmqR3s1/g== --osd-uuid bf76eb5b-4443-4fef-9329-f9115f67fcec 2023-08-16T02:11:23.271 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.268+0000 7fcf4bb1b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:23.301 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.299+0000 7fcf4bb1b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:23.302 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.300+0000 7fcf4bb1b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:23.303 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.300+0000 7fcf4bb1b640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-08-16T02:11:23.303 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.300+0000 7fcf4bb1b640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-08-16T02:11:23.303 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.300+0000 7fcf4bb1b640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2023-08-16T02:11:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:23.366+0000 7fcf4bb1b640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2023-08-16T02:11:28.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2023-08-16T02:11:28.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:11:28.828 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd1 key to auth repository 2023-08-16T02:11:28.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2023-08-16T02:11:28.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:11:29.228 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2023-08-16T02:11:29.229 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.1 2023-08-16T02:11:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:11:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:11:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:11:29.259 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:29.257+0000 7ff5c7549640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:29.260 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:29.259+0000 7ff5c7549640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:29.261 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:29.260+0000 7ff5c7549640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:29.564 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2023-08-16T02:11:29.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:11:29.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2023-08-16T02:11:29.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:11:29.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:11:29.565 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:29.566 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:11:29.566 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:11:29.566 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:29.567 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:11:29.787 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:29.786+0000 7ff5c7549640 -1 Falling back to public interface 2023-08-16T02:11:29.908 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:30.909 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:30.910 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:30.910 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:11:30.911 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:11:30.911 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:30.912 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:11:31.233 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:32.163 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:32.161+0000 7ff5c7549640 -1 osd.1 0 log_to_monitors true 2023-08-16T02:11:32.234 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:32.235 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:32.235 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:11:32.235 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:11:32.235 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:32.236 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:11:32.608 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:33.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:33.609 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:33.610 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:11:33.610 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:11:33.610 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:33.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2023-08-16T02:11:33.923 INFO:tasks.workunit.client.0.smithi150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4222999041,v1:127.0.0.1:6811/4222999041] [v2:127.0.0.1:6812/4222999041,v1:127.0.0.1:6813/4222999041] exists,up bf76eb5b-4443-4fef-9329-f9115f67fcec 2023-08-16T02:11:33.927 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:11:33.927 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:11:33.927 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:11:33.927 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:33.927 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:33.928 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:11:33.928 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:11:33.928 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2023-08-16T02:11:33.928 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:11:33.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2023-08-16T02:11:33.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:11:33.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:11:33.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:11:33.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:11:33.929 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:33.930 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:11:33.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:11:33.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:11:33.933 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:11:33.933 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2023-08-16T02:11:33.933 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:11:33.933 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6ef3adb-cfa9-4596-8201-54ef09d2b0d5 2023-08-16T02:11:33.933 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f6ef3adb-cfa9-4596-8201-54ef09d2b0d5' 2023-08-16T02:11:33.933 INFO:tasks.workunit.client.0.smithi150.stdout:add osd2 f6ef3adb-cfa9-4596-8201-54ef09d2b0d5 2023-08-16T02:11:33.934 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:11:33.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVMNxkuIg8OBAAqVbEfAEtNqdKnBNy25UAZA== 2023-08-16T02:11:33.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVMNxkuIg8OBAAqVbEfAEtNqdKnBNy25UAZA=="}' 2023-08-16T02:11:33.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6ef3adb-cfa9-4596-8201-54ef09d2b0d5 -i td/osd-recovery-scrub/2/new.json 2023-08-16T02:11:34.361 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:11:34.377 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2023-08-16T02:11:34.377 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBVMNxkuIg8OBAAqVbEfAEtNqdKnBNy25UAZA== --osd-uuid f6ef3adb-cfa9-4596-8201-54ef09d2b0d5 2023-08-16T02:11:34.406 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.405+0000 7f6fc1556640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:34.437 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.436+0000 7f6fc1556640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:34.438 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.436+0000 7f6fc1556640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:34.439 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.437+0000 7f6fc1556640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-08-16T02:11:34.439 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.437+0000 7f6fc1556640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-08-16T02:11:34.439 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.437+0000 7f6fc1556640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2023-08-16T02:11:34.499 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:34.497+0000 7f6fc1556640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2023-08-16T02:11:40.525 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2023-08-16T02:11:40.525 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:11:40.525 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd2 key to auth repository 2023-08-16T02:11:40.526 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2023-08-16T02:11:40.526 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:11:40.930 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.2 2023-08-16T02:11:40.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2023-08-16T02:11:40.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:11:40.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:40.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:11:40.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:11:40.961 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:40.959+0000 7fc1fa4ec640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:40.963 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:40.961+0000 7fc1fa4ec640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:40.963 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:40.962+0000 7fc1fa4ec640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:41.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2023-08-16T02:11:41.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:11:41.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2023-08-16T02:11:41.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:11:41.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:11:41.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:41.276 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:11:41.277 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:11:41.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:41.277 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:11:41.486 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:41.485+0000 7fc1fa4ec640 -1 Falling back to public interface 2023-08-16T02:11:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:42.644 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:42.644 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:42.645 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:11:42.645 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:11:42.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:42.646 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:11:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:43.855 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:43.853+0000 7fc1fa4ec640 -1 osd.2 0 log_to_monitors true 2023-08-16T02:11:43.965 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:43.965 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:43.966 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:11:43.966 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:11:43.966 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:43.966 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:11:44.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:45.287 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:45.288 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:45.288 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:11:45.288 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:11:45.289 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:45.289 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2023-08-16T02:11:45.647 INFO:tasks.workunit.client.0.smithi150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2114775159,v1:127.0.0.1:6819/2114775159] [v2:127.0.0.1:6820/2114775159,v1:127.0.0.1:6821/2114775159] exists,up f6ef3adb-cfa9-4596-8201-54ef09d2b0d5 2023-08-16T02:11:45.653 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:11:45.653 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:11:45.653 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:11:45.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:45.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:11:45.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:11:45.656 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:11:45.656 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:11:45.656 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2023-08-16T02:11:45.656 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2023-08-16T02:11:45.656 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:11:45.656 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:45.657 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:11:45.658 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:11:45.659 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2023-08-16T02:11:45.660 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:11:45.660 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=76e94974-4d83-4a71-a00d-1f41292f07f9 2023-08-16T02:11:45.660 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 76e94974-4d83-4a71-a00d-1f41292f07f9' 2023-08-16T02:11:45.660 INFO:tasks.workunit.client.0.smithi150.stdout:add osd3 76e94974-4d83-4a71-a00d-1f41292f07f9 2023-08-16T02:11:45.661 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:11:45.669 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhMNxkRPPCJxAAZd50SB8f251R4kQUAbDz0Q== 2023-08-16T02:11:45.669 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhMNxkRPPCJxAAZd50SB8f251R4kQUAbDz0Q=="}' 2023-08-16T02:11:45.669 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 76e94974-4d83-4a71-a00d-1f41292f07f9 -i td/osd-recovery-scrub/3/new.json 2023-08-16T02:11:46.047 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:11:46.060 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2023-08-16T02:11:46.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBhMNxkRPPCJxAAZd50SB8f251R4kQUAbDz0Q== --osd-uuid 76e94974-4d83-4a71-a00d-1f41292f07f9 2023-08-16T02:11:46.093 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.091+0000 7f005a30b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:46.165 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.164+0000 7f005a30b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:46.166 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.165+0000 7f005a30b640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:46.167 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.165+0000 7f005a30b640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-08-16T02:11:46.167 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.165+0000 7f005a30b640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-08-16T02:11:46.168 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.165+0000 7f005a30b640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2023-08-16T02:11:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:46.239+0000 7f005a30b640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2023-08-16T02:11:52.541 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2023-08-16T02:11:52.541 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:11:52.541 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2023-08-16T02:11:52.542 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd3 key to auth repository 2023-08-16T02:11:52.542 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:11:52.944 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2023-08-16T02:11:52.944 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.3 2023-08-16T02:11:52.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:52.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:11:52.945 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:11:52.946 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:11:52.974 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:52.972+0000 7f3d0b22a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:52.976 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:52.974+0000 7f3d0b22a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:52.977 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:52.975+0000 7f3d0b22a640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:53.258 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2023-08-16T02:11:53.258 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:11:53.258 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2023-08-16T02:11:53.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:11:53.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:11:53.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:53.259 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:11:53.259 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:11:53.260 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:53.260 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:11:53.499 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:53.497+0000 7f3d0b22a640 -1 Falling back to public interface 2023-08-16T02:11:53.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:54.612 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:54.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:54.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:11:54.613 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:11:54.614 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:54.614 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:11:54.934 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:55.843 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:55.840+0000 7f3d0b22a640 -1 osd.3 0 log_to_monitors true 2023-08-16T02:11:55.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:55.936 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:55.936 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:11:55.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:11:55.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:55.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:11:56.272 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:11:57.273 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:11:57.273 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:11:57.274 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:11:57.274 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:11:57.274 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:11:57.274 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2023-08-16T02:11:57.621 INFO:tasks.workunit.client.0.smithi150.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3136981720,v1:127.0.0.1:6827/3136981720] [v2:127.0.0.1:6828/3136981720,v1:127.0.0.1:6829/3136981720] exists,up 76e94974-4d83-4a71-a00d-1f41292f07f9 2023-08-16T02:11:57.627 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:11:57.628 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:11:57.628 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:11:57.629 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:11:57.629 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:11:57.629 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:11:57.629 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:11:57.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2023-08-16T02:11:57.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:11:57.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2023-08-16T02:11:57.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:11:57.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:11:57.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:11:57.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:11:57.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2023-08-16T02:11:57.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2023-08-16T02:11:57.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:11:57.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:11:57.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:11:57.633 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:11:57.633 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:11:57.633 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:11:57.633 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:11:57.633 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:11:57.634 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:11:57.635 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2023-08-16T02:11:57.636 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:11:57.636 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac9674a1-191f-4e62-b59a-4014f14b1d52 2023-08-16T02:11:57.636 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 ac9674a1-191f-4e62-b59a-4014f14b1d52' 2023-08-16T02:11:57.636 INFO:tasks.workunit.client.0.smithi150.stdout:add osd4 ac9674a1-191f-4e62-b59a-4014f14b1d52 2023-08-16T02:11:57.637 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:11:57.642 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtMNxkzDMxJhAArDZXVs8MymlqDac4XTF3dw== 2023-08-16T02:11:57.643 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtMNxkzDMxJhAArDZXVs8MymlqDac4XTF3dw=="}' 2023-08-16T02:11:57.643 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac9674a1-191f-4e62-b59a-4014f14b1d52 -i td/osd-recovery-scrub/4/new.json 2023-08-16T02:11:58.011 INFO:tasks.workunit.client.0.smithi150.stdout:4 2023-08-16T02:11:58.023 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2023-08-16T02:11:58.025 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBtMNxkzDMxJhAArDZXVs8MymlqDac4XTF3dw== --osd-uuid ac9674a1-191f-4e62-b59a-4014f14b1d52 2023-08-16T02:11:58.058 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.055+0000 7fda87b22640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:58.087 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.085+0000 7fda87b22640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:58.088 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.086+0000 7fda87b22640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:11:58.089 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.086+0000 7fda87b22640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-08-16T02:11:58.089 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.086+0000 7fda87b22640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-08-16T02:11:58.089 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.086+0000 7fda87b22640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2023-08-16T02:11:58.169 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:11:58.166+0000 7fda87b22640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2023-08-16T02:12:03.931 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2023-08-16T02:12:03.932 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:12:03.933 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2023-08-16T02:12:03.934 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd4 key to auth repository 2023-08-16T02:12:03.934 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:12:04.338 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2023-08-16T02:12:04.339 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.4 2023-08-16T02:12:04.339 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:12:04.339 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:04.340 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:12:04.340 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:12:04.369 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:04.366+0000 7ffa7c84d640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:04.387 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:04.384+0000 7ffa7c84d640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:04.388 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:04.385+0000 7ffa7c84d640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:04.653 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2023-08-16T02:12:04.653 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:12:04.653 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2023-08-16T02:12:04.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:12:04.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:12:04.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:04.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:12:04.654 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:12:04.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:04.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-08-16T02:12:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:04.918+0000 7ffa7c84d640 -1 Falling back to public interface 2023-08-16T02:12:04.995 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:05.997 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:05.997 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:05.997 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:12:05.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:12:05.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:05.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-08-16T02:12:06.318 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:07.291 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:07.289+0000 7ffa7c84d640 -1 osd.4 0 log_to_monitors true 2023-08-16T02:12:07.318 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:07.319 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:07.319 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:12:07.319 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:12:07.320 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:07.320 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-08-16T02:12:07.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:08.750 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:08.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:08.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:12:08.751 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:12:08.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:08.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2023-08-16T02:12:09.071 INFO:tasks.workunit.client.0.smithi150.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4115354865,v1:127.0.0.1:6835/4115354865] [v2:127.0.0.1:6836/4115354865,v1:127.0.0.1:6837/4115354865] exists,up ac9674a1-191f-4e62-b59a-4014f14b1d52 2023-08-16T02:12:09.074 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:12:09.074 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:12:09.074 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:12:09.074 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:12:09.074 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:12:09.075 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:12:09.076 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:12:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:12:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:12:09.079 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:12:09.079 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:12:09.079 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2023-08-16T02:12:09.079 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:12:09.079 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3ef54b5-6695-467d-bf2a-1f94978c9ea2 2023-08-16T02:12:09.080 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 f3ef54b5-6695-467d-bf2a-1f94978c9ea2' 2023-08-16T02:12:09.080 INFO:tasks.workunit.client.0.smithi150.stdout:add osd5 f3ef54b5-6695-467d-bf2a-1f94978c9ea2 2023-08-16T02:12:09.080 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:12:09.093 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5MNxkK2tsBRAA0lCOYay1raZCKSRMXG1sVw== 2023-08-16T02:12:09.093 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5MNxkK2tsBRAA0lCOYay1raZCKSRMXG1sVw=="}' 2023-08-16T02:12:09.093 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3ef54b5-6695-467d-bf2a-1f94978c9ea2 -i td/osd-recovery-scrub/5/new.json 2023-08-16T02:12:09.510 INFO:tasks.workunit.client.0.smithi150.stdout:5 2023-08-16T02:12:09.523 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2023-08-16T02:12:09.524 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB5MNxkK2tsBRAA0lCOYay1raZCKSRMXG1sVw== --osd-uuid f3ef54b5-6695-467d-bf2a-1f94978c9ea2 2023-08-16T02:12:09.556 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.553+0000 7f7c99b40640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:09.595 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.592+0000 7f7c99b40640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.593+0000 7f7c99b40640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.594+0000 7f7c99b40640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-08-16T02:12:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.594+0000 7f7c99b40640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-08-16T02:12:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.594+0000 7f7c99b40640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2023-08-16T02:12:09.671 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:09.668+0000 7f7c99b40640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2023-08-16T02:12:15.683 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2023-08-16T02:12:15.684 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:12:15.684 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2023-08-16T02:12:15.685 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd5 key to auth repository 2023-08-16T02:12:15.686 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:12:16.090 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2023-08-16T02:12:16.090 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.5 2023-08-16T02:12:16.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:12:16.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:16.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:12:16.091 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:12:16.121 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:16.119+0000 7fa601465640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:16.123 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:16.121+0000 7fa601465640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:16.124 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:16.122+0000 7fa601465640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:16.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2023-08-16T02:12:16.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:12:16.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2023-08-16T02:12:16.437 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:12:16.438 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:12:16.438 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:16.438 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:12:16.438 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:12:16.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:16.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-08-16T02:12:16.650 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:16.649+0000 7fa601465640 -1 Falling back to public interface 2023-08-16T02:12:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:17.766 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:17.767 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:17.767 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:12:17.767 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:12:17.768 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:17.768 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-08-16T02:12:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:19.010 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:19.009+0000 7fa601465640 -1 osd.5 0 log_to_monitors true 2023-08-16T02:12:19.093 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:19.093 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:19.093 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:12:19.094 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:12:19.094 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:19.095 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-08-16T02:12:19.418 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:20.419 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:20.419 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:20.420 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:12:20.420 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:12:20.421 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:20.421 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2023-08-16T02:12:20.743 INFO:tasks.workunit.client.0.smithi150.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1346423214,v1:127.0.0.1:6843/1346423214] [v2:127.0.0.1:6844/1346423214,v1:127.0.0.1:6845/1346423214] exists,up f3ef54b5-6695-467d-bf2a-1f94978c9ea2 2023-08-16T02:12:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:12:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:12:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:12:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:12:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:12:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:12:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:12:20.750 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:12:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:12:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:12:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:12:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2023-08-16T02:12:20.753 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:12:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1898c57a-d3cb-47ed-a129-2a4a9e02ae8d 2023-08-16T02:12:20.753 INFO:tasks.workunit.client.0.smithi150.stdout:add osd6 1898c57a-d3cb-47ed-a129-2a4a9e02ae8d 2023-08-16T02:12:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 1898c57a-d3cb-47ed-a129-2a4a9e02ae8d' 2023-08-16T02:12:20.754 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:12:20.765 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEMNxkEzR9LRAA03wvmOV2eO+nso9TOq9+HA== 2023-08-16T02:12:20.766 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEMNxkEzR9LRAA03wvmOV2eO+nso9TOq9+HA=="}' 2023-08-16T02:12:20.766 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1898c57a-d3cb-47ed-a129-2a4a9e02ae8d -i td/osd-recovery-scrub/6/new.json 2023-08-16T02:12:21.145 INFO:tasks.workunit.client.0.smithi150.stdout:6 2023-08-16T02:12:21.157 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2023-08-16T02:12:21.158 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCEMNxkEzR9LRAA03wvmOV2eO+nso9TOq9+HA== --osd-uuid 1898c57a-d3cb-47ed-a129-2a4a9e02ae8d 2023-08-16T02:12:21.191 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.189+0000 7fefa4b09640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:21.262 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.257+0000 7fefa4b09640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:21.263 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.262+0000 7fefa4b09640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:21.264 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.262+0000 7fefa4b09640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-08-16T02:12:21.264 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.262+0000 7fefa4b09640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-08-16T02:12:21.264 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.262+0000 7fefa4b09640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2023-08-16T02:12:21.405 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:21.403+0000 7fefa4b09640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2023-08-16T02:12:27.201 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2023-08-16T02:12:27.201 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:12:27.202 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2023-08-16T02:12:27.203 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd6 key to auth repository 2023-08-16T02:12:27.203 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:12:27.611 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2023-08-16T02:12:27.612 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.6 2023-08-16T02:12:27.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:27.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:12:27.613 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:12:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:12:27.643 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:27.641+0000 7fa1f9c3c640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:27.645 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:27.644+0000 7fa1f9c3c640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:27.646 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:27.644+0000 7fa1f9c3c640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:27.963 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2023-08-16T02:12:27.963 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2023-08-16T02:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:12:27.965 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:12:27.965 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:27.965 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-08-16T02:12:28.181 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:28.180+0000 7fa1f9c3c640 -1 Falling back to public interface 2023-08-16T02:12:28.345 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:29.347 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:29.347 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:29.347 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:12:29.347 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:12:29.348 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:29.348 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-08-16T02:12:29.668 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:30.554 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:30.552+0000 7fa1f9c3c640 -1 osd.6 0 log_to_monitors true 2023-08-16T02:12:30.669 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:30.670 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:30.670 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:12:30.670 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:12:30.671 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:30.671 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-08-16T02:12:30.996 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:31.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:31.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:31.998 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:12:31.998 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:12:32.000 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:32.000 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2023-08-16T02:12:32.358 INFO:tasks.workunit.client.0.smithi150.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4288581499,v1:127.0.0.1:6851/4288581499] [v2:127.0.0.1:6852/4288581499,v1:127.0.0.1:6853/4288581499] exists,up 1898c57a-d3cb-47ed-a129-2a4a9e02ae8d 2023-08-16T02:12:32.361 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:12:32.361 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:12:32.361 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:12:32.361 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2023-08-16T02:12:32.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2023-08-16T02:12:32.363 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:12:32.364 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2023-08-16T02:12:32.365 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2023-08-16T02:12:32.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2023-08-16T02:12:32.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2023-08-16T02:12:32.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2023-08-16T02:12:32.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2023-08-16T02:12:32.366 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2023-08-16T02:12:32.366 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2023-08-16T02:12:32.367 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4586675d-5b5e-4325-b13d-5f48e379c67a 2023-08-16T02:12:32.367 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 4586675d-5b5e-4325-b13d-5f48e379c67a' 2023-08-16T02:12:32.367 INFO:tasks.workunit.client.0.smithi150.stdout:add osd7 4586675d-5b5e-4325-b13d-5f48e379c67a 2023-08-16T02:12:32.368 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2023-08-16T02:12:32.380 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQMNxk+JyJFhAA/7EbNu2ViTuIP/tUGpf+zQ== 2023-08-16T02:12:32.380 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQMNxk+JyJFhAA/7EbNu2ViTuIP/tUGpf+zQ=="}' 2023-08-16T02:12:32.380 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4586675d-5b5e-4325-b13d-5f48e379c67a -i td/osd-recovery-scrub/7/new.json 2023-08-16T02:12:32.815 INFO:tasks.workunit.client.0.smithi150.stdout:7 2023-08-16T02:12:32.827 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2023-08-16T02:12:32.828 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCQMNxk+JyJFhAA/7EbNu2ViTuIP/tUGpf+zQ== --osd-uuid 4586675d-5b5e-4325-b13d-5f48e379c67a 2023-08-16T02:12:32.862 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.860+0000 7febcfa75640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:32.891 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.890+0000 7febcfa75640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:32.892 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.891+0000 7febcfa75640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:32.893 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.891+0000 7febcfa75640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-08-16T02:12:32.893 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.891+0000 7febcfa75640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-08-16T02:12:32.893 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.891+0000 7febcfa75640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2023-08-16T02:12:32.967 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:32.965+0000 7febcfa75640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2023-08-16T02:12:38.899 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2023-08-16T02:12:38.900 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2023-08-16T02:12:38.900 INFO:tasks.workunit.client.0.smithi150.stdout:adding osd7 key to auth repository 2023-08-16T02:12:38.901 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2023-08-16T02:12:38.901 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2023-08-16T02:12:39.303 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2023-08-16T02:12:39.304 INFO:tasks.workunit.client.0.smithi150.stdout:start osd.7 2023-08-16T02:12:39.304 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=2752a73a-b786-417b-836d-274629efc135 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.109770/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2023-08-16T02:12:39.305 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2023-08-16T02:12:39.305 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2023-08-16T02:12:39.305 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2023-08-16T02:12:39.334 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:39.332+0000 7f15ac113640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:39.336 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:39.334+0000 7f15ac113640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:39.337 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:39.335+0000 7f15ac113640 -1 WARNING: all dangerous and experimental features are enabled. 2023-08-16T02:12:39.650 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2023-08-16T02:12:39.650 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2023-08-16T02:12:39.650 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2023-08-16T02:12:39.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2023-08-16T02:12:39.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2023-08-16T02:12:39.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:39.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2023-08-16T02:12:39.651 INFO:tasks.workunit.client.0.smithi150.stdout:0 2023-08-16T02:12:39.652 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:39.652 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-08-16T02:12:39.875 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:39.873+0000 7f15ac113640 -1 Falling back to public interface 2023-08-16T02:12:39.965 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:40.967 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:40.967 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:40.967 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2023-08-16T02:12:40.967 INFO:tasks.workunit.client.0.smithi150.stdout:1 2023-08-16T02:12:40.968 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:40.968 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-08-16T02:12:41.289 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:42.265 INFO:tasks.workunit.client.0.smithi150.stderr:2023-08-16T02:12:42.262+0000 7f15ac113640 -1 osd.7 0 log_to_monitors true 2023-08-16T02:12:42.291 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:42.291 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:42.292 INFO:tasks.workunit.client.0.smithi150.stdout:2 2023-08-16T02:12:42.292 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2023-08-16T02:12:42.293 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:42.293 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-08-16T02:12:42.689 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2023-08-16T02:12:43.690 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2023-08-16T02:12:43.691 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2023-08-16T02:12:43.691 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2023-08-16T02:12:43.691 INFO:tasks.workunit.client.0.smithi150.stdout:3 2023-08-16T02:12:43.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2023-08-16T02:12:43.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2023-08-16T02:12:44.043 INFO:tasks.workunit.client.0.smithi150.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3244920453,v1:127.0.0.1:6859/3244920453] [v2:127.0.0.1:6860/3244920453,v1:127.0.0.1:6861/3244920453] exists,up 4586675d-5b5e-4325-b13d-5f48e379c67a 2023-08-16T02:12:44.044 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2023-08-16T02:12:44.044 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2023-08-16T02:12:44.044 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2023-08-16T02:12:44.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2023-08-16T02:12:44.045 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2023-08-16T02:12:44.479 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'test' created 2023-08-16T02:12:44.488 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2023-08-16T02:12:45.492 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2023-08-16T02:12:45.492 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2023-08-16T02:12:45.492 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2023-08-16T02:12:45.492 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2023-08-16T02:12:45.492 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2023-08-16T02:12:45.492 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2023-08-16T02:12:45.493 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2023-08-16T02:12:45.493 INFO:tasks.workunit.client.0.smithi150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2023-08-16T02:12:45.493 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2023-08-16T02:12:45.493 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2023-08-16T02:12:45.493 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2023-08-16T02:12:45.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2023-08-16T02:12:45.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2023-08-16T02:12:45.593 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2023-08-16T02:12:45.594 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2023-08-16T02:12:45.594 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2023-08-16T02:12:46.110 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2023-08-16T02:12:46.110 INFO:tasks.workunit.client.0.smithi150.stderr:1 2023-08-16T02:12:46.110 INFO:tasks.workunit.client.0.smithi150.stderr:2 2023-08-16T02:12:46.110 INFO:tasks.workunit.client.0.smithi150.stderr:3 2023-08-16T02:12:46.110 INFO:tasks.workunit.client.0.smithi150.stderr:4 2023-08-16T02:12:46.111 INFO:tasks.workunit.client.0.smithi150.stderr:5 2023-08-16T02:12:46.111 INFO:tasks.workunit.client.0.smithi150.stderr:6 2023-08-16T02:12:46.111 INFO:tasks.workunit.client.0.smithi150.stderr:7' 2023-08-16T02:12:46.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2023-08-16T02:12:46.111 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:46.111 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2023-08-16T02:12:46.328 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836498 2023-08-16T02:12:46.328 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836498 2023-08-16T02:12:46.329 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498' 2023-08-16T02:12:46.329 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:46.329 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2023-08-16T02:12:46.522 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2023-08-16T02:12:46.523 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2023-08-16T02:12:46.523 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2023-08-16T02:12:46.523 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:46.523 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2023-08-16T02:12:46.815 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509454 2023-08-16T02:12:46.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509454 2023-08-16T02:12:46.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2023-08-16T02:12:46.816 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:46.816 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2023-08-16T02:12:47.008 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345931 2023-08-16T02:12:47.009 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345931 2023-08-16T02:12:47.009 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931' 2023-08-16T02:12:47.009 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:47.009 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2023-08-16T02:12:47.135 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=107374182409 2023-08-16T02:12:47.136 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 107374182409 2023-08-16T02:12:47.136 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409' 2023-08-16T02:12:47.136 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:47.136 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2023-08-16T02:12:47.285 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=128849018887 2023-08-16T02:12:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 128849018887 2023-08-16T02:12:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887' 2023-08-16T02:12:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:47.287 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2023-08-16T02:12:47.480 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=150323855365 2023-08-16T02:12:47.480 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 150323855365 2023-08-16T02:12:47.480 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2023-08-16T02:12:47.480 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2023-08-16T02:12:47.480 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2023-08-16T02:12:47.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=171798691842 2023-08-16T02:12:47.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 171798691842 2023-08-16T02:12:47.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2023-08-16T02:12:47.631 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:47.631 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836498 2023-08-16T02:12:47.631 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:47.632 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2023-08-16T02:12:47.632 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836498 2023-08-16T02:12:47.632 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:47.633 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836498 2023-08-16T02:12:47.633 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2023-08-16T02:12:47.633 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.0 seq 21474836498 2023-08-16T02:12:47.633 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2023-08-16T02:12:47.947 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836498 2023-08-16T02:12:47.948 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:47.949 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2023-08-16T02:12:47.949 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:47.949 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2023-08-16T02:12:47.950 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2023-08-16T02:12:47.950 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:47.950 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2023-08-16T02:12:47.951 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2023-08-16T02:12:47.951 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.1 seq 42949672976 2023-08-16T02:12:47.951 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-08-16T02:12:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672975 -lt 42949672976 2023-08-16T02:12:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2023-08-16T02:12:49.293 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2023-08-16T02:12:49.293 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2023-08-16T02:12:49.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2023-08-16T02:12:49.618 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:49.619 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509454 2023-08-16T02:12:49.619 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:49.619 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2023-08-16T02:12:49.620 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509454 2023-08-16T02:12:49.620 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:49.620 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509454 2023-08-16T02:12:49.621 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2023-08-16T02:12:49.621 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.2 seq 64424509454 2023-08-16T02:12:49.622 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2023-08-16T02:12:49.969 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509454 -lt 64424509454 2023-08-16T02:12:49.969 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:49.970 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345931 2023-08-16T02:12:49.970 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:49.970 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2023-08-16T02:12:49.971 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345931 2023-08-16T02:12:49.971 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:49.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345931 2023-08-16T02:12:49.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2023-08-16T02:12:49.972 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.3 seq 85899345931 2023-08-16T02:12:49.973 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2023-08-16T02:12:50.284 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345932 -lt 85899345931 2023-08-16T02:12:50.284 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:50.284 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 4-107374182409 2023-08-16T02:12:50.285 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:50.285 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=4 2023-08-16T02:12:50.285 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 4-107374182409 2023-08-16T02:12:50.286 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:50.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=107374182409 2023-08-16T02:12:50.286 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2023-08-16T02:12:50.287 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.4 seq 107374182409 2023-08-16T02:12:50.287 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 4 2023-08-16T02:12:50.603 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 107374182410 -lt 107374182409 2023-08-16T02:12:50.603 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:50.603 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 5-128849018887 2023-08-16T02:12:50.604 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:50.604 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=5 2023-08-16T02:12:50.604 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 5-128849018887 2023-08-16T02:12:50.604 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:50.604 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=128849018887 2023-08-16T02:12:50.605 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2023-08-16T02:12:50.605 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.5 seq 128849018887 2023-08-16T02:12:50.605 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 5 2023-08-16T02:12:50.954 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 128849018887 -lt 128849018887 2023-08-16T02:12:50.954 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:50.955 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 6-150323855365 2023-08-16T02:12:50.955 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:50.955 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=6 2023-08-16T02:12:50.956 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 6-150323855365 2023-08-16T02:12:50.956 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:50.956 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=150323855365 2023-08-16T02:12:50.957 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2023-08-16T02:12:50.957 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.6 seq 150323855365 2023-08-16T02:12:50.957 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 6 2023-08-16T02:12:51.297 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 150323855365 -lt 150323855365 2023-08-16T02:12:51.297 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2023-08-16T02:12:51.297 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 7-171798691842 2023-08-16T02:12:51.298 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2023-08-16T02:12:51.298 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=7 2023-08-16T02:12:51.298 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 7-171798691842 2023-08-16T02:12:51.299 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2023-08-16T02:12:51.299 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=171798691842 2023-08-16T02:12:51.299 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2023-08-16T02:12:51.300 INFO:tasks.workunit.client.0.smithi150.stdout:waiting osd.7 seq 171798691842 2023-08-16T02:12:51.300 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 7 2023-08-16T02:12:51.641 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 171798691843 -lt 171798691842 2023-08-16T02:12:51.642 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2023-08-16T02:12:51.642 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-08-16T02:12:51.643 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-08-16T02:12:52.054 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2023-08-16T02:12:52.054 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2023-08-16T02:12:52.055 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2023-08-16T02:12:52.055 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2023-08-16T02:12:52.055 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2023-08-16T02:12:52.055 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2023-08-16T02:12:52.056 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2023-08-16T02:12:52.056 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2023-08-16T02:12:52.364 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2023-08-16T02:12:52.364 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2023-08-16T02:12:52.365 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2023-08-16T02:12:52.365 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2023-08-16T02:12:52.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2023-08-16T02:12:52.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2023-08-16T02:12:52.753 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2023-08-16T02:12:52.754 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2023-08-16T02:12:52.754 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2023-08-16T02:12:52.754 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2023-08-16T02:12:53.094 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2023-08-16T02:12:53.094 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.109770 bs=1M count=50 2023-08-16T02:12:53.323 INFO:tasks.workunit.client.0.smithi150.stderr:50+0 records in 2023-08-16T02:12:53.324 INFO:tasks.workunit.client.0.smithi150.stderr:50+0 records out 2023-08-16T02:12:53.324 INFO:tasks.workunit.client.0.smithi150.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.228968 s, 229 MB/s 2023-08-16T02:12:53.325 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2023-08-16T02:12:53.325 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:12:53.325 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.109770 2023-08-16T02:12:54.905 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:12:54.905 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.109770 2023-08-16T02:12:56.361 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:12:56.362 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.109770 2023-08-16T02:12:58.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:12:58.061 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.109770 2023-08-16T02:12:59.668 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:12:59.668 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.109770 2023-08-16T02:13:01.166 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:01.167 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.109770 2023-08-16T02:13:02.807 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:02.807 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.109770 2023-08-16T02:13:04.322 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:04.322 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.109770 2023-08-16T02:13:05.846 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.109770 2023-08-16T02:13:07.453 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:07.453 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.109770 2023-08-16T02:13:08.885 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:08.885 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.109770 2023-08-16T02:13:10.411 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:10.411 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.109770 2023-08-16T02:13:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.109770 2023-08-16T02:13:13.507 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:13.507 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.109770 2023-08-16T02:13:15.147 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:15.148 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.109770 2023-08-16T02:13:16.782 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:16.782 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.109770 2023-08-16T02:13:18.347 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:18.347 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.109770 2023-08-16T02:13:19.937 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:19.938 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.109770 2023-08-16T02:13:21.453 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:21.453 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.109770 2023-08-16T02:13:22.985 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:22.985 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.109770 2023-08-16T02:13:24.850 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:24.850 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.109770 2023-08-16T02:13:26.349 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:26.349 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.109770 2023-08-16T02:13:28.006 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:28.006 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.109770 2023-08-16T02:13:29.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:29.630 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.109770 2023-08-16T02:13:31.194 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:31.195 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.109770 2023-08-16T02:13:32.772 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:32.772 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.109770 2023-08-16T02:13:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.109770 2023-08-16T02:13:35.941 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:35.942 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.109770 2023-08-16T02:13:37.490 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:37.490 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.109770 2023-08-16T02:13:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:38.997 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.109770 2023-08-16T02:13:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:40.596 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.109770 2023-08-16T02:13:42.143 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:42.144 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.109770 2023-08-16T02:13:43.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:43.843 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.109770 2023-08-16T02:13:45.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:45.275 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.109770 2023-08-16T02:13:46.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:46.974 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.109770 2023-08-16T02:13:48.480 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:48.481 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.109770 2023-08-16T02:13:49.971 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:49.971 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.109770 2023-08-16T02:13:51.587 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:51.587 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.109770 2023-08-16T02:13:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.109770 2023-08-16T02:13:54.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2023-08-16T02:13:54.692 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.109770 2023-08-16T02:13:56.191 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.109770 2023-08-16T02:13:56.197 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2023-08-16T02:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 1 size to 3 2023-08-16T02:13:56.694 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2023-08-16T02:13:56.997 INFO:tasks.workunit.client.0.smithi150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2023-08-16T02:13:56.997 INFO:tasks.workunit.client.0.smithi150.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2023-08-16T02:12:46.372704+0000 48'26 48:55 [5] 5 [5] 5 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.997 INFO:tasks.workunit.client.0.smithi150.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.371994+0000 48'13 48:42 [3] 3 [3] 3 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.372686+0000 48'13 48:31 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.330987+0000 48'13 48:31 [0] 0 [0] 0 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.331331+0000 48'13 48:31 [0] 0 [0] 0 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:46.372174+0000 0'0 48:18 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.268225+0000 48'13 48:31 [3] 3 [3] 3 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:46.268239+0000 0'0 48:17 [5] 5 [5] 5 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.998 INFO:tasks.workunit.client.0.smithi150.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.154854+0000 48'13 48:31 [0] 0 [0] 0 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:45.910313+0000 0'0 48:17 [6] 6 [6] 6 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2023-08-16T02:12:46.268192+0000 48'39 48:55 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:45.778119+0000 0'0 48:19 [2] 2 [2] 2 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.748454+0000 48'13 48:30 [4] 4 [4] 4 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.12 2 0 0 0 0 81788928 0 0 20 0 20 active+clean 2023-08-16T02:12:45.331345+0000 48'20 48:38 [0] 0 [0] 0 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.154939+0000 48'13 48:32 [0] 0 [0] 0 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:56.999 INFO:tasks.workunit.client.0.smithi150.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.372441+0000 48'13 48:31 [5] 5 [5] 5 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2023-08-16T02:12:45.888799+0000 48'39 48:60 [4] 4 [4] 4 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2023-08-16T02:12:46.372415+0000 48'52 48:74 [3] 3 [3] 3 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2023-08-16T02:12:45.888740+0000 48'52 48:74 [4] 4 [4] 4 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.331200+0000 48'13 48:38 [0] 0 [0] 0 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.699544+0000 48'13 48:37 [7] 7 [7] 7 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.372580+0000 48'13 48:42 [5] 5 [5] 5 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.000 INFO:tasks.workunit.client.0.smithi150.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:45.910461+0000 0'0 48:21 [6] 6 [6] 6 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.778054+0000 48'13 48:34 [6] 6 [6] 6 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2023-08-16T02:12:46.268187+0000 48'26 48:45 [5] 5 [5] 5 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:46.372571+0000 0'0 48:20 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.268319+0000 48'13 48:33 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.372287+0000 48'13 48:33 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2023-08-16T02:12:46.372031+0000 0'0 48:20 [3] 3 [3] 3 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.699541+0000 48'13 48:27 [7] 7 [7] 7 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.001 INFO:tasks.workunit.client.0.smithi150.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:46.268188+0000 48'13 48:30 [1] 1 [1] 1 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.002 INFO:tasks.workunit.client.0.smithi150.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2023-08-16T02:12:45.910440+0000 48'13 48:31 [6] 6 [6] 6 0'0 2023-08-16T02:12:44.422235+0000 0'0 2023-08-16T02:12:44.422235+0000 0 0 periodic scrub scheduled @ 2023-08-17T02:12:44.422235+0000 0 0 2023-08-16T02:13:57.002 INFO:tasks.workunit.client.0.smithi150.stdout: 2023-08-16T02:13:57.002 INFO:tasks.workunit.client.0.smithi150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2023-08-16T02:13:57.002 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:13:57.009 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2023-08-16T02:13:57.127 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:13:57.128 INFO:tasks.workunit.client.0.smithi150.stdout: "osd_recovery_max_active": "0" 2023-08-16T02:13:57.128 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:13:57.138 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2023-08-16T02:13:57.321 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:13:57.321 INFO:tasks.workunit.client.0.smithi150.stdout: "osd_recovery_max_active_hdd": "3" 2023-08-16T02:13:57.321 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:13:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2023-08-16T02:13:57.665 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2023-08-16T02:13:57.665 INFO:tasks.workunit.client.0.smithi150.stdout: "osd_recovery_max_active_ssd": "10" 2023-08-16T02:13:57.665 INFO:tasks.workunit.client.0.smithi150.stdout:} 2023-08-16T02:13:57.675 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2023-08-16T02:13:57.676 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:13:57.676 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:13:57.676 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:13:57.677 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:13:57.677 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:13:58.131 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:13:58.141 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2023-08-16T02:13:58.141 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:00.142 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2023-08-16T02:14:00.143 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2023-08-16T02:14:00.143 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2023-08-16T02:14:00.144 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:00.145 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:00.145 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:00.145 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:00.146 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:00.456 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:00.466 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2023-08-16T02:14:00.467 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:02.468 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2023-08-16T02:14:02.468 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2023-08-16T02:14:02.469 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2023-08-16T02:14:02.470 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:02.470 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:02.470 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:02.471 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:02.471 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:02.780 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:02.790 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2023-08-16T02:14:02.790 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:04.792 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2023-08-16T02:14:04.792 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2023-08-16T02:14:04.793 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2023-08-16T02:14:04.793 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:04.794 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:04.794 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:04.795 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:04.795 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:05.113 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:05.113 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:07.114 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 3 -eq 10 2023-08-16T02:14:07.115 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 3 + 1 2023-08-16T02:14:07.116 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=4 2023-08-16T02:14:07.116 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:07.117 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:07.117 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:07.118 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:07.118 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:07.428 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:07.438 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:07.439 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:09.440 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 4 -eq 10 2023-08-16T02:14:09.440 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 4 + 1 2023-08-16T02:14:09.441 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=5 2023-08-16T02:14:09.441 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:09.442 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:09.442 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:09.443 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:09.443 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:09.990 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:10.000 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:10.000 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 5 -eq 10 2023-08-16T02:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 5 + 1 2023-08-16T02:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=6 2023-08-16T02:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:12.311 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:12.321 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:12.321 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:14.322 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 6 -eq 10 2023-08-16T02:14:14.323 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 6 + 1 2023-08-16T02:14:14.323 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=7 2023-08-16T02:14:14.324 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:14.325 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:14.326 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:14.326 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:14.326 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:14.637 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:16.649 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 7 -eq 10 2023-08-16T02:14:16.649 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 7 + 1 2023-08-16T02:14:16.650 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=8 2023-08-16T02:14:16.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:16.652 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:16.652 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:16.652 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:16.653 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:16.961 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:16.971 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:16.972 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:18.973 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 8 -eq 10 2023-08-16T02:14:18.973 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 8 + 1 2023-08-16T02:14:18.974 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=9 2023-08-16T02:14:18.974 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:18.975 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:18.975 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:18.976 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:18.976 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:19.283 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:19.293 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:19.293 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:21.294 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 9 -eq 10 2023-08-16T02:14:21.295 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 9 + 1 2023-08-16T02:14:21.296 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=10 2023-08-16T02:14:21.296 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2023-08-16T02:14:21.297 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2023-08-16T02:14:21.297 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2023-08-16T02:14:21.297 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2023-08-16T02:14:21.298 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2023-08-16T02:14:21.641 INFO:tasks.workunit.client.0.smithi150.stderr:dumped pgs 2023-08-16T02:14:21.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2023-08-16T02:14:21.651 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2023-08-16T02:14:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 10 -eq 10 2023-08-16T02:14:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:293: TEST_recovery_scrub_2: echo 'Not enough recovery started simultaneously' 2023-08-16T02:14:23.653 INFO:tasks.workunit.client.0.smithi150.stdout:Not enough recovery started simultaneously 2023-08-16T02:14:23.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:294: TEST_recovery_scrub_2: return 1 2023-08-16T02:14:23.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: return 1 2023-08-16T02:14:23.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: main: code=1 2023-08-16T02:14:23.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/osd-recovery-scrub 1 2023-08-16T02:14:23.654 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2023-08-16T02:14:23.655 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2023-08-16T02:14:23.770 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2023-08-16T02:14:23.770 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2023-08-16T02:14:23.771 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2023-08-16T02:14:23.771 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2023-08-16T02:14:23.772 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2023-08-16T02:14:23.772 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2023-08-16T02:14:23.773 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2023-08-16T02:14:23.774 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:14:23.774 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2023-08-16T02:14:23.775 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2023-08-16T02:14:23.775 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-16T02:14:23.775 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:14:23.776 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2023-08-16T02:14:23.776 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2023-08-16T02:14:23.777 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2023-08-16T02:14:23.777 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-recovery-scrub/mgr.x.log td/osd-recovery-scrub/mon.a.log td/osd-recovery-scrub/osd.0.log td/osd-recovery-scrub/osd.1.log td/osd-recovery-scrub/osd.2.log td/osd-recovery-scrub/osd.3.log td/osd-recovery-scrub/osd.4.log td/osd-recovery-scrub/osd.5.log td/osd-recovery-scrub/osd.6.log td/osd-recovery-scrub/osd.7.log /home/ubuntu/cephtest/archive/log 2023-08-16T02:14:23.796 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2023-08-16T02:14:23.805 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2023-08-16T02:14:23.805 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2023-08-16T02:14:23.805 INFO:tasks.workunit.client.0.smithi150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.109770 2023-08-16T02:14:23.805 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.109770 2023-08-16T02:14:23.806 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2023-08-16T02:14:23.806 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2023-08-16T02:14:23.806 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 1 2023-08-16T02:14:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T02:14:23.808 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2023-08-16T02:14:23.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-08-16T02:14:24.081 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5259484dbc61e8573f27e692560c6c107b9d4c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh' 2023-08-16T02:14:24.146 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c2551b1fc1594233a5c4750a46696a21 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5259484dbc61e8573f27e692560c6c107b9d4c1/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5259484dbc61e8573f27e692560c6c107b9d4c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh' 2023-08-16T02:14:24.149 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-08-16T02:14:24.166 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-08-16T02:14:24.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-16T02:14:34.242 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-08-16T02:14:34.251 INFO:teuthology.task.clock:Checking final clock skew... 2023-08-16T02:14:34.252 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-16T02:14:34.271 INFO:teuthology.orchestra.run.smithi150.stderr:bash: ntpq: command not found 2023-08-16T02:14:34.276 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-16T02:14:34.276 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2023-08-16T02:14:34.277 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 62 +2275us[+2302us] +/- 54ms 2023-08-16T02:14:34.277 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -2090us[-2062us] +/- 66ms 2023-08-16T02:14:34.277 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -807us[ -781us] +/- 79ms 2023-08-16T02:14:34.277 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-08-16T02:14:34.278 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-08-16T02:14:34.287 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-08-16T02:14:34.288 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-08-16T02:14:34.296 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-16T02:14:34.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-16T02:14:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T02:14:34.461 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-08-16T02:14:34.472 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-08-16T02:14:34.485 INFO:teuthology.task.internal:Duration was 1446.835073 seconds 2023-08-16T02:14:34.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-08-16T02:14:34.496 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-08-16T02:14:34.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-16T02:14:34.549 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-16T02:14:34.639 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-08-16T02:14:34.640 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2023-08-16T02:14:34.640 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-16T02:14:34.662 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-08-16T02:14:34.662 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-16T02:14:34.739 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-08-16T02:14:34.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-16T02:14:34.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-08-16T02:14:34.865 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-16T02:14:34.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-16T02:14:34.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-08-16T02:14:34.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:14:34.958 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2023-08-16T02:14:34.973 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-16T02:14:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T02:14:35.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-08-16T02:14:35.040 INFO:teuthology.task.internal:Transferring archived files... 2023-08-16T02:14:35.041 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369212/remote/smithi150 2023-08-16T02:14:35.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-16T02:14:36.480 INFO:teuthology.task.internal:Removing archive directory... 2023-08-16T02:14:36.480 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-16T02:14:36.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-08-16T02:14:36.537 INFO:teuthology.task.internal:Not uploading archives. 2023-08-16T02:14:36.537 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-08-16T02:14:36.591 INFO:teuthology.task.internal:Tidying up after the test... 2023-08-16T02:14:36.591 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-16T02:14:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: 262170 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:14 /home/ubuntu/cephtest 2023-08-16T02:14:36.608 INFO:teuthology.orchestra.run.smithi150.stdout: 659002 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 16 02:03 /home/ubuntu/cephtest/mnt.0 2023-08-16T02:14:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: 659003 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 16 02:07 /home/ubuntu/cephtest/mnt.0/client.0 2023-08-16T02:14:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: 1052522 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 16 02:13 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-16T02:14:36.609 INFO:teuthology.orchestra.run.smithi150.stdout: 1052523 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 16 02:14 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2023-08-16T02:14:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-16T02:14:36.610 INFO:teuthology.orchestra.run.smithi150.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-08-16T02:14:36.611 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-08-16T02:14:36.614 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-08-16T02:14:36.630 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-16T02:14:36.647 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi150 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-08-16T02:14:36.647 INFO:teuthology.nuke:Checking targets against current locks 2023-08-16T02:14:37.525 DEBUG:teuthology.nuke:shortname: smithi150 2023-08-16T02:14:37.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-16T02:14:37.547 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 01:43:22.341376', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-16T02:14:37.595 INFO:teuthology.orchestra.console.smithi150:Power off 2023-08-16T02:14:37.597 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:14:37.622 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2023-08-16T02:14:37.623 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:14:37.646 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2023-08-16T02:14:41.647 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:14:42.759 DEBUG:teuthology.orchestra.console.smithi150:check power output: 2023-08-16T02:14:46.761 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:14:46.872 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2023-08-16T02:14:46.973 INFO:teuthology.orchestra.console.smithi150:Power off completed 2023-08-16T02:14:47.080 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} duration: 1446.8350732326508 failure_reason: 'Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi150 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b5259484dbc61e8573f27e692560c6c107b9d4c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c2551b1fc1594233a5c4750a46696a21 status: fail success: false 2023-08-16T02:14:47.081 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-16T02:14:50.577 INFO:teuthology.run:FAIL