2024-02-09T22:03:31.508 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-09T22:03:31.508 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381 --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} -- /home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381/orig.config.yaml 2024-02-09T22:03:31.530 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T22:03:31.607 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381 branch: main description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7554381' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2011 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBPPP9QvWd6JLIkaxOmMoCvhRQXYAPEWnAsQ1wjtiF/mqHI8zoRxfvUWu53BaX+tgKxG3Jt9aeNOXXt6sNf+3Q= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:35:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-09T22:03:31.607 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-09T22:03:31.608 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-09T22:03:31.608 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-09T22:03:31.610 INFO:teuthology.task.internal:Checking packages... 2024-02-09T22:03:31.637 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-09T22:03:31.638 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T22:03:31.638 INFO:teuthology.packaging:ref: None 2024-02-09T22:03:31.638 INFO:teuthology.packaging:tag: None 2024-02-09T22:03:31.638 INFO:teuthology.packaging:branch: main 2024-02-09T22:03:31.638 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:03:31.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-09T22:03:31.901 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-09T22:03:31.903 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-09T22:03:31.909 INFO:teuthology.task.internal:no buildpackages task found 2024-02-09T22:03:31.909 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-09T22:03:31.940 INFO:teuthology.task.internal:Saving configuration 2024-02-09T22:03:31.949 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-09T22:03:32.001 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T22:03:32.026 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 21:58:38.728637', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBPPP9QvWd6JLIkaxOmMoCvhRQXYAPEWnAsQ1wjtiF/mqHI8zoRxfvUWu53BaX+tgKxG3Jt9aeNOXXt6sNf+3Q='} 2024-02-09T22:03:32.026 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-09T22:03:32.032 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-02-09T22:03:32.032 INFO:teuthology.run_tasks:Running task console_log... 2024-02-09T22:03:32.079 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1f1ae13f70>, signals=[15]) 2024-02-09T22:03:32.080 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-09T22:03:32.086 INFO:teuthology.task.internal:Opening connections... 2024-02-09T22:03:32.086 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-02-09T22:03:32.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:03:32.179 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-09T22:03:32.185 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-02-09T22:03:32.248 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-02-09T22:03:32.249 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-02-09T22:03:32.305 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-09T22:03:32.306 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T22:03:32.307 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-02-09T22:03:32.330 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-09T22:03:32.341 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-09T22:03:32.353 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-09T22:03:32.354 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-02-09T22:03:32.369 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-09T22:03:32.375 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-09T22:03:32.375 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-02-09T22:03:32.534 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-09T22:03:32.565 INFO:teuthology.run_tasks:Running task kernel... 2024-02-09T22:03:32.613 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-09T22:03:32.613 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-09T22:03:32.613 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T22:03:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:03:32.631 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-02-09T22:03:32.689 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-02-09T22:03:32.689 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-02-09T22:03:32.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-02-09T22:03:34.887 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-09T22:03:36.320 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-02-09T22:03:37.652 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 8.0 MB/s | 8.0 MB 00:01 2024-02-09T22:03:39.748 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-02-09T22:03:45.604 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 5.9 MB 00:02 2024-02-09T22:03:47.071 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-02-09T22:03:47.981 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-02-09T22:03:52.822 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-02-09T22:03:55.096 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-09T22:03:55.096 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-09T22:03:55.192 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:03:55.201 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-02-09T22:03:55.202 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:03:55.203 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-02-09T22:03:55.203 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 122 M 2024-02-09T22:03:55.203 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:03:55.565 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-09T22:03:56.099 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-09T22:03:56.874 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 23 MB/s | 39 MB 00:01 2024-02-09T22:03:57.232 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 20 MB/s | 33 MB 00:01 2024-02-09T22:03:57.236 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:03:57.239 INFO:teuthology.orchestra.run.smithi134.stdout:Total 49 MB/s | 99 MB 00:02 2024-02-09T22:03:57.239 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:03:57.611 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:03:57.612 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:03:58.665 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:03:58.665 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:04:01.065 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:04:01.744 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-09T22:04:01.874 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T22:04:04.617 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T22:04:04.810 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T22:04:09.540 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T22:04:09.655 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:04:14.395 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:04:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:04:43.681 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-09T22:04:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:04:43.753 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:04:44.172 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:04:44.173 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-09T22:04:44.173 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T22:04:44.173 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T22:04:44.585 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:04:44.585 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:04:44.585 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:04:44.585 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-09T22:04:44.586 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-09T22:04:44.586 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-09T22:04:44.586 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-09T22:04:44.586 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:04:44.586 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:04:45.128 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T22:04:45.318 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T22:04:45.318 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T22:04:45.318 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-09T22:04:45.319 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T22:04:45.319 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T22:04:45.319 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-09T22:04:45.319 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T22:04:45.319 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T22:04:45.427 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T22:04:45.428 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T22:04:45.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-02-09T22:04:45.463 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-09T22:04:45.464 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:04:45.465 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T22:04:45.466 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T22:04:45.467 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-419.el9.x86_64 2024-02-09T22:04:45.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T22:04:46.059 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-02-09T22:04:47.491 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-09T22:04:47.512 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-02-09T22:04:47.514 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-02-09T22:04:47.530 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.d8PtQ3eVjo 2024-02-09T22:04:47.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.d8PtQ3eVjo 2024-02-09T22:04:47.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.d8PtQ3eVjo 2024-02-09T22:04:47.748 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.d8PtQ3eVjo is 6KB 2024-02-09T22:04:47.796 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.d8PtQ3eVjo 2024-02-09T22:04:47.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T22:04:47.875 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-09T22:04:47.875 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-09T22:04:47.876 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-09T22:04:47.876 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-09T22:04:47.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-09T22:04:48.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-02-09T22:05:18.045 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-09T22:05:18.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-09T22:05:18.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:06:18.109 DEBUG:teuthology.orchestra.remote:timed out 2024-02-09T22:06:27.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-09T22:06:27.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:06:27.536 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-02-09T22:06:28.102 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-09T22:06:28.102 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-09T22:06:28.103 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-02-09T22:06:28.117 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-419.el9.x86_64 2024-02-09T22:06:28.117 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-09T22:06:28.117 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T22:06:28.118 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T22:06:29.118 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-09T22:06:29.118 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-09T22:06:29.197 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-02-09T22:06:29.237 DEBUG:teuthology.parallel:result is None 2024-02-09T22:06:29.237 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-09T22:06:29.245 INFO:teuthology.task.internal:Creating test directory... 2024-02-09T22:06:29.245 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T22:06:29.260 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-09T22:06:29.302 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-09T22:06:29.347 INFO:teuthology.task.internal:Creating archive directory... 2024-02-09T22:06:29.348 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T22:06:29.367 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-09T22:06:29.391 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-09T22:06:29.392 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T22:06:29.443 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:06:29.456 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:06:29.458 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-09T22:06:29.465 INFO:teuthology.task.internal:Configuring sudo... 2024-02-09T22:06:29.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T22:06:29.525 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-09T22:06:29.534 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-09T22:06:29.534 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T22:06:29.579 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:06:29.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:06:29.720 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:06:29.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:06:29.845 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:06:29.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T22:06:29.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-02-09T22:06:29.998 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T22:06:30.322 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-09T22:06:30.331 INFO:teuthology.task.internal:Starting timer... 2024-02-09T22:06:30.331 INFO:teuthology.run_tasks:Running task pcp... 2024-02-09T22:06:30.364 INFO:teuthology.run_tasks:Running task selinux... 2024-02-09T22:06:30.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-02-09T22:06:30.449 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-02-09T22:06:30.451 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-09T22:06:30.452 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-02-09T22:06:30.512 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-02-09T22:06:30.513 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-02-09T22:06:30.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T22:06:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:06:30.581 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-09T22:06:30.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-02-09T22:06:30.660 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-09T22:06:30.673 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-09T22:06:30.680 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-09T22:06:30.819 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-09T22:06:30.836 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-09T22:06:30.838 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-09T22:10:38.473 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-02-09T22:10:38.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-09T22:10:38.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:10:38.565 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-02-09T22:10:38.647 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-09T22:10:38.647 INFO:teuthology.run_tasks:Running task clock... 2024-02-09T22:10:38.659 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-09T22:10:38.659 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T22:10:38.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T22:10:38.718 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T22:10:38.735 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T22:10:38.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-02-09T22:10:38.778 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-02-09T22:10:38.795 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T22:10:38.811 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T22:10:38.864 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-02-09T22:10:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T22:10:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-02-09T22:10:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:10:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:10:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:10:38.867 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:10:38.868 INFO:teuthology.run_tasks:Running task install... 2024-02-09T22:10:38.877 DEBUG:teuthology.task.install:project ceph 2024-02-09T22:10:38.877 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-09T22:10:38.877 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-09T22:10:38.878 INFO:teuthology.task.install:Using flavor: default 2024-02-09T22:10:38.883 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-09T22:10:38.883 INFO:teuthology.task.install:extra packages: [] 2024-02-09T22:10:38.884 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-09T22:10:38.884 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:10:39.046 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-09T22:10:39.047 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-09T22:10:39.308 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T22:10:39.308 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:10:39.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T22:10:39.337 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-09T22:10:39.338 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T22:10:39.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T22:10:39.486 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T22:10:39.552 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-02-09T22:10:39.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-02-09T22:10:39.793 INFO:teuthology.orchestra.run.smithi134.stdout:74 files removed 2024-02-09T22:10:39.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-02-09T22:10:40.746 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 134 kB/s | 82 kB 00:00 2024-02-09T22:10:41.230 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-02-09T22:10:41.606 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-02-09T22:10:42.192 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-09T22:10:43.559 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-02-09T22:10:44.585 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-02-09T22:10:46.642 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-02-09T22:10:50.810 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 5.9 MB 00:00 2024-02-09T22:10:52.761 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-02-09T22:10:55.837 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 20 MB 00:02 2024-02-09T22:11:00.729 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-09T22:11:03.365 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:11:03.368 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:11:03.368 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:11:03.368 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-09T22:11:03.369 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-09T22:11:03.370 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:11:03.371 INFO:teuthology.orchestra.run.smithi134.stdout:Install 30 Packages 2024-02-09T22:11:03.372 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-02-09T22:11:03.372 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:03.372 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 58 M 2024-02-09T22:11:03.372 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:11:04.196 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 7.1 MB/s | 5.3 MB 00:00 2024-02-09T22:11:04.271 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 336 kB/s | 25 kB 00:00 2024-02-09T22:11:04.380 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 6.7 MB/s | 743 kB 00:00 2024-02-09T22:11:04.489 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): libradosstriper1-19.0.0-1361.g290212a1. 4.5 MB/s | 499 kB 00:00 2024-02-09T22:11:05.264 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): ceph-common-19.0.0-1361.g290212a1.el9.x 11 MB/s | 20 MB 00:01 2024-02-09T22:11:05.339 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): librgw2-19.0.0-1361.g290212a1.el9.x86_6 6.0 MB/s | 5.1 MB 00:00 2024-02-09T22:11:05.364 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-ceph-argparse-19.0.0-1361.g2902 451 kB/s | 45 kB 00:00 2024-02-09T22:11:05.390 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-ceph-common-19.0.0-1361.g290212 2.6 MB/s | 131 kB 00:00 2024-02-09T22:11:05.415 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-cephfs-19.0.0-1361.g290212a1.el 3.1 MB/s | 161 kB 00:00 2024-02-09T22:11:05.524 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): ceph-radosgw-19.0.0-1361.g290212a1.el9 4.7 MB/s | 9.7 MB 00:02 2024-02-09T22:11:05.549 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-rados-19.0.0-1361.g290212a1.el 2.0 MB/s | 319 kB 00:00 2024-02-09T22:11:05.574 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-rbd-19.0.0-1361.g290212a1.el9. 1.8 MB/s | 298 kB 00:00 2024-02-09T22:11:05.625 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): python3-rgw-19.0.0-1361.g290212a1.el9. 989 kB/s | 99 kB 00:00 2024-02-09T22:11:05.691 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2024-02-09T22:11:05.708 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2024-02-09T22:11:05.750 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-02-09T22:11:05.776 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-02-09T22:11:05.801 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-02-09T22:11:05.826 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 554 kB/s | 41 kB 00:00 2024-02-09T22:11:05.851 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 677 kB/s | 34 kB 00:00 2024-02-09T22:11:05.877 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 215 kB 00:00 2024-02-09T22:11:05.902 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-02-09T22:11:06.036 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-02-09T22:11:06.119 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-02-09T22:11:06.145 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-02-09T22:11:06.170 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-02-09T22:11:06.237 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 838 kB 00:00 2024-02-09T22:11:06.262 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-02-09T22:11:06.362 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-02-09T22:11:06.421 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-02-09T22:11:06.613 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): librados2-19.0.0-1361.g290212a1.el9.x8 9.7 MB/s | 3.4 MB 00:00 2024-02-09T22:11:06.671 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-02-09T22:11:06.677 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:11:06.677 INFO:teuthology.orchestra.run.smithi134.stdout:Total 17 MB/s | 58 MB 00:03 2024-02-09T22:11:06.677 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:11:06.709 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:11:06.709 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:11:07.178 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:11:07.178 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:11:08.656 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:11:08.738 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-09T22:11:08.768 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-09T22:11:08.793 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-09T22:11:08.816 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-09T22:11:08.832 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-09T22:11:09.006 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-09T22:11:09.063 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-09T22:11:09.400 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-09T22:11:09.454 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-09T22:11:09.537 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-09T22:11:09.626 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-09T22:11:09.655 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-09T22:11:09.703 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-09T22:11:09.738 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-09T22:11:09.782 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-09T22:11:09.851 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-09T22:11:09.874 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-09T22:11:09.936 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-09T22:11:09.964 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-09T22:11:10.066 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-09T22:11:10.099 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-09T22:11:10.154 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-09T22:11:10.180 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-09T22:11:10.210 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-09T22:11:10.242 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-09T22:11:10.560 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-09T22:11:10.608 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-09T22:11:10.678 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-09T22:11:10.710 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-09T22:11:10.755 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-09T22:11:11.212 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-09T22:11:11.233 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-09T22:11:11.304 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-09T22:11:11.380 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-09T22:11:11.419 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-09T22:11:13.582 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-09T22:11:13.615 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-09T22:11:14.164 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-09T22:11:14.202 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-09T22:11:14.291 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-09T22:11:14.291 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-09T22:11:14.292 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-09T22:11:14.292 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:14.328 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-09T22:11:23.888 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-09T22:11:23.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-02-09T22:11:23.889 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:24.807 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-09T22:11:24.832 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-09T22:11:24.832 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:11:24.832 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-09T22:11:24.832 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:11:24.832 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:11:24.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:24.835 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-09T22:11:24.927 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-09T22:11:24.927 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-09T22:11:26.083 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-09T22:11:26.083 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-09T22:11:26.083 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-09T22:11:26.083 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-09T22:11:26.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-09T22:11:26.085 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-09T22:11:26.086 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-09T22:11:26.086 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-09T22:11:26.086 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-09T22:11:26.086 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-09T22:11:26.086 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-09T22:11:26.087 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-09T22:11:26.087 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-09T22:11:26.087 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.183 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.184 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-02-09T22:11:27.185 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-09T22:11:27.186 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:27.186 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:11:27.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-02-09T22:11:28.027 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:28 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:11:28.184 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:11:28.185 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:11:28.185 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:11:28.186 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:11:28.186 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:11:28.186 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-09T22:11:28.186 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:11:28.186 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:11:28.187 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-02-09T22:11:28.188 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:28.188 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-02-09T22:11:28.189 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 204 M 2024-02-09T22:11:28.189 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:11:28.560 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 804 kB/s | 300 kB 00:00 2024-02-09T22:11:28.644 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 753 kB/s | 63 kB 00:00 2024-02-09T22:11:28.744 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 186 kB 00:00 2024-02-09T22:11:28.778 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 276 kB/s | 163 kB 00:00 2024-02-09T22:11:28.828 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-02-09T22:11:43.565 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 3.1 MB/s | 47 MB 00:15 2024-02-09T22:11:43.565 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:11:43.565 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.1 MB/s | 48 MB 00:15 2024-02-09T22:11:43.565 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:11:43.583 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:11:43.584 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:11:43.985 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:11:43.985 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:11:44.562 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:11:44.625 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-09T22:11:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-09T22:11:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-09T22:11:44.752 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-09T22:11:44.780 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-09T22:11:49.270 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-09T22:11:49.345 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-09T22:11:51.752 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-09T22:11:51.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-09T22:11:51.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-09T22:11:51.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-09T22:11:51.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-09T22:11:51.753 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-09T22:11:52.115 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:52.116 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:11:52.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-02-09T22:11:53.032 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:53 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:11:53.204 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:11:53.206 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-02-09T22:11:53.206 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:11:53.206 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-02-09T22:11:53.206 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-09T22:11:53.207 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-09T22:11:53.208 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-09T22:11:53.209 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-02-09T22:11:53.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:11:53.211 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-02-09T22:11:53.211 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 107 M 2024-02-09T22:11:53.211 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:11:53.870 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-02-09T22:11:55.695 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 724 kB/s | 1.5 MB 00:02 2024-02-09T22:11:56.445 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 760 kB/s | 2.1 MB 00:02 2024-02-09T22:11:57.004 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1361.g2902 443 kB/s | 247 kB 00:00 2024-02-09T22:11:57.329 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 575 kB/s | 187 kB 00:00 2024-02-09T22:11:57.454 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-02-09T22:11:57.497 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 764 kB/s | 32 kB 00:00 2024-02-09T22:11:57.546 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-02-09T22:11:57.613 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 579 kB/s | 39 kB 00:00 2024-02-09T22:11:57.813 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2024-02-09T22:11:58.455 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 1.8 MB/s | 1.1 MB 00:00 2024-02-09T22:11:58.556 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-02-09T22:11:58.664 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 124 kB 00:00 2024-02-09T22:11:58.824 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 708 kB/s | 113 kB 00:00 2024-02-09T22:11:58.956 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.4 MB/s | 187 kB 00:00 2024-02-09T22:11:59.173 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-02-09T22:11:59.407 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 648 kB/s | 151 kB 00:00 2024-02-09T22:11:59.457 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 874 kB/s | 43 kB 00:00 2024-02-09T22:11:59.549 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-02-09T22:11:59.683 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-02-09T22:11:59.741 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-02-09T22:11:59.800 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-02-09T22:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 465 kB/s | 23 kB 00:00 2024-02-09T22:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-02-09T22:11:59.959 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-02-09T22:12:00.018 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 795 kB/s | 46 kB 00:00 2024-02-09T22:12:00.077 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-02-09T22:12:00.152 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-02-09T22:12:00.219 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-02-09T22:12:00.327 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 831 kB/s | 90 kB 00:00 2024-02-09T22:12:00.428 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86 731 kB/s | 4.7 MB 00:06 2024-02-09T22:12:00.453 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 286 kB/s | 36 kB 00:00 2024-02-09T22:12:00.553 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-02-09T22:12:00.629 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 267 kB/s | 20 kB 00:00 2024-02-09T22:12:00.812 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 598 kB/s | 230 kB 00:00 2024-02-09T22:12:05.087 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 1.8 MB/s | 17 MB 00:09 2024-02-09T22:12:05.091 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:05.091 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.5 MB/s | 30 MB 00:11 2024-02-09T22:12:05.112 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:05.156 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:05.156 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:05.559 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:05.560 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:06.911 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:06.973 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-09T22:12:07.005 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-09T22:12:07.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-09T22:12:07.105 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-09T22:12:07.148 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-09T22:12:07.181 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-09T22:12:07.274 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-09T22:12:07.353 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-09T22:12:07.401 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-09T22:12:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-09T22:12:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-09T22:12:07.565 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-09T22:12:07.613 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-09T22:12:07.669 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-09T22:12:07.856 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-09T22:12:07.902 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-09T22:12:07.948 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-09T22:12:07.983 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-09T22:12:08.013 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-09T22:12:08.089 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-09T22:12:08.132 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-09T22:12:08.165 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-09T22:12:08.206 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-09T22:12:08.241 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-09T22:12:08.284 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-09T22:12:08.373 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-09T22:12:08.414 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-09T22:12:08.487 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-09T22:12:08.623 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-09T22:12:08.740 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-09T22:12:08.907 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-09T22:12:08.935 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-09T22:12:08.962 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-09T22:12:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:12:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-09T22:12:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:12:08.962 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:12:08.962 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:10.386 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-09T22:12:10.412 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-09T22:12:10.412 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:12:10.415 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-09T22:12:10.415 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:12:10.415 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:12:10.415 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:10.854 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-09T22:12:10.877 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-09T22:12:10.878 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:12:10.878 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-09T22:12:10.878 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:12:10.878 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:12:10.878 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:11.088 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-09T22:12:11.111 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-09T22:12:11.111 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:12:11.111 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-09T22:12:11.111 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:12:11.111 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:12:11.111 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:11.176 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-09T22:12:12.925 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-09T22:12:12.925 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-09T22:12:12.925 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-09T22:12:12.925 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-09T22:12:12.925 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-09T22:12:12.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-09T22:12:12.927 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-09T22:12:12.928 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-09T22:12:12.928 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-09T22:12:12.928 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-09T22:12:12.928 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-09T22:12:12.928 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-09T22:12:12.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-09T22:12:12.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-09T22:12:13.963 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-09T22:12:13.963 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:13.963 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:13.963 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:13.963 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:13.963 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-09T22:12:13.964 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-09T22:12:13.965 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:13.966 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:14.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-02-09T22:12:14.967 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:14 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:15.073 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:12:15.125 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:15.127 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:12:15.127 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:15.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-02-09T22:12:15.800 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:15 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:15.960 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:15.961 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:15.961 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:15.961 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:15.961 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:15.961 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-09T22:12:15.961 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 748 k 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 754 k 2024-02-09T22:12:15.962 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:16.761 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 936 kB/s | 748 kB 00:00 2024-02-09T22:12:16.761 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:16.762 INFO:teuthology.orchestra.run.smithi134.stdout:Total 935 kB/s | 748 kB 00:00 2024-02-09T22:12:16.762 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:16.766 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:16.767 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:16.772 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:16.773 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:16.849 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:17.267 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-09T22:12:17.340 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-09T22:12:17.764 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-09T22:12:18.061 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-09T22:12:18.061 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:18.061 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:18.061 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:18.061 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:18.061 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:18.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-02-09T22:12:18.912 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:18 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:19.076 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:19.076 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 436 k 2024-02-09T22:12:19.077 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:19.472 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 369 kB/s | 145 kB 00:00 2024-02-09T22:12:19.473 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:19.473 INFO:teuthology.orchestra.run.smithi134.stdout:Total 367 kB/s | 145 kB 00:00 2024-02-09T22:12:19.473 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:19.482 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:19.482 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:19.536 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:19.536 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:19.687 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:19.740 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-09T22:12:19.766 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-09T22:12:19.766 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:12:19.766 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-09T22:12:19.766 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:20.827 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-09T22:12:20.828 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:20.828 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:20.828 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:20.828 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:20.828 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:21.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-02-09T22:12:21.660 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:21 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:21.767 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:12:21.818 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:21.819 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:12:21.819 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:21.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-02-09T22:12:22.481 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:22 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:22.641 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:12:22.642 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-02-09T22:12:22.643 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-02-09T22:12:22.644 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:22.644 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-02-09T22:12:22.644 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-02-09T22:12:22.644 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:23.184 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1361.g290 66 kB/s | 15 kB 00:00 2024-02-09T22:12:23.209 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1361.g29 109 kB/s | 28 kB 00:00 2024-02-09T22:12:23.384 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-02-09T22:12:23.418 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-02-09T22:12:23.460 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 3.5 MB/s | 1.8 MB 00:00 2024-02-09T22:12:23.686 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-02-09T22:12:23.786 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 60 kB/s | 19 kB 00:00 2024-02-09T22:12:23.928 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-02-09T22:12:24.020 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-02-09T22:12:24.045 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 558 kB/s | 144 kB 00:00 2024-02-09T22:12:24.120 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-02-09T22:12:24.287 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-02-09T22:12:24.292 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:24.292 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.1 MB/s | 6.8 MB 00:01 2024-02-09T22:12:24.310 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:24.326 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:24.326 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:24.495 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:24.495 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:25.008 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:25.070 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-09T22:12:25.114 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-09T22:12:25.161 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-09T22:12:25.191 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-09T22:12:25.288 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-09T22:12:25.416 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-09T22:12:25.502 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-09T22:12:25.545 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-09T22:12:25.574 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-09T22:12:25.604 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-09T22:12:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-09T22:12:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-09T22:12:26.743 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-09T22:12:26.743 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-09T22:12:26.743 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-09T22:12:26.743 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-09T22:12:26.743 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-09T22:12:26.744 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-09T22:12:27.113 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-09T22:12:27.114 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-09T22:12:27.114 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-09T22:12:27.114 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-09T22:12:27.114 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:27.114 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:27.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-09T22:12:27.979 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:27 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:28.139 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:28.140 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-02-09T22:12:28.140 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-09T22:12:28.141 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-02-09T22:12:28.142 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:28.143 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-02-09T22:12:28.143 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-02-09T22:12:28.143 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:28.709 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 165 kB/s | 32 kB 00:00 2024-02-09T22:12:28.759 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 343 kB/s | 17 kB 00:00 2024-02-09T22:12:28.818 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 588 kB/s | 34 kB 00:00 2024-02-09T22:12:29.051 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.6 MB/s | 3.0 MB 00:00 2024-02-09T22:12:29.435 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.7 MB/s | 4.7 MB 00:00 2024-02-09T22:12:29.502 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-02-09T22:12:29.527 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-09T22:12:29.786 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 802 kB 00:00 2024-02-09T22:12:29.961 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-02-09T22:12:29.986 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 981 kB/s | 194 kB 00:00 2024-02-09T22:12:30.178 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-02-09T22:12:32.162 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.0 MB/s | 7.4 MB 00:03 2024-02-09T22:12:32.165 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:32.165 INFO:teuthology.orchestra.run.smithi134.stdout:Total 9.2 MB/s | 37 MB 00:04 2024-02-09T22:12:32.180 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:32.208 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:32.208 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:32.572 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:32.572 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:33.285 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:33.368 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-09T22:12:33.399 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-09T22:12:33.492 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-09T22:12:33.527 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-09T22:12:33.707 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-09T22:12:33.734 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-09T22:12:33.811 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-09T22:12:34.484 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-09T22:12:34.588 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-09T22:12:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-09T22:12:36.845 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-09T22:12:36.875 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-09T22:12:39.462 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-09T22:12:39.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-09T22:12:39.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-09T22:12:39.462 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-09T22:12:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-09T22:12:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-09T22:12:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-09T22:12:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-09T22:12:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-09T22:12:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-09T22:12:39.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-09T22:12:39.464 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-09T22:12:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-09T22:12:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:39.918 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:39.918 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:39.919 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:40.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-02-09T22:12:40.964 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:40 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:41.124 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:41.126 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-09T22:12:41.126 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:41.126 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-09T22:12:41.126 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-09T22:12:41.127 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:41.128 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:41.129 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-09T22:12:41.129 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-02-09T22:12:41.129 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:41.129 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-02-09T22:12:41.130 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-02-09T22:12:41.130 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:41.698 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 142 kB/s | 24 kB 00:00 2024-02-09T22:12:41.723 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 92 kB/s | 18 kB 00:00 2024-02-09T22:12:41.890 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 915 kB/s | 174 kB 00:00 2024-02-09T22:12:41.957 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 121 kB/s | 52 kB 00:00 2024-02-09T22:12:41.982 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 511 kB/s | 132 kB 00:00 2024-02-09T22:12:42.024 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 211 kB 00:00 2024-02-09T22:12:42.050 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 464 kB/s | 43 kB 00:00 2024-02-09T22:12:42.608 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 51 kB/s | 32 kB 00:00 2024-02-09T22:12:42.925 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-google-auth-2.27.0-1.el9.noarch 232 kB/s | 203 kB 00:00 2024-02-09T22:12:43.217 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 203 kB/s | 59 kB 00:00 2024-02-09T22:12:43.384 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-02-09T22:12:43.501 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.1 MB/s | 1.0 MB 00:00 2024-02-09T22:12:43.634 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 9.3 kB/s | 15 kB 00:01 2024-02-09T22:12:43.639 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:43.640 INFO:teuthology.orchestra.run.smithi134.stdout:Total 835 kB/s | 2.0 MB 00:02 2024-02-09T22:12:43.640 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:43.654 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:43.654 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:43.762 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:43.762 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:44.277 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:44.439 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-09T22:12:44.493 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-09T22:12:44.548 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-09T22:12:44.577 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-09T22:12:44.619 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-09T22:12:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-09T22:12:44.813 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-09T22:12:44.870 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-09T22:12:45.581 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-09T22:12:45.625 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-09T22:12:45.660 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-09T22:12:45.708 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-09T22:12:45.729 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-09T22:12:46.566 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-09T22:12:46.566 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-09T22:12:46.566 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-09T22:12:46.567 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-09T22:12:46.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:46.961 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:47.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-02-09T22:12:47.833 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:47 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:47.996 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:47.996 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-09T22:12:47.996 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-09T22:12:47.997 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-02-09T22:12:47.998 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:47.998 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-02-09T22:12:47.998 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-02-09T22:12:47.998 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:48.310 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-09T22:12:48.493 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-02-09T22:12:48.519 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 371 kB/s | 138 kB 00:00 2024-02-09T22:12:48.852 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-02-09T22:12:48.877 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 142 kB/s | 51 kB 00:00 2024-02-09T22:12:48.927 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 889 kB/s | 548 kB 00:00 2024-02-09T22:12:48.929 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:48.929 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-02-09T22:12:48.930 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:48.938 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:48.938 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:49.027 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:49.029 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:49.328 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:49.499 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-09T22:12:49.553 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-09T22:12:50.066 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-09T22:12:50.147 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-09T22:12:50.209 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-09T22:12:50.232 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-09T22:12:51.026 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-09T22:12:51.026 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-09T22:12:51.026 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-09T22:12:51.026 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-09T22:12:51.026 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-09T22:12:51.026 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-09T22:12:51.348 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-09T22:12:51.349 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:51.349 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:51.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-02-09T22:12:52.218 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:52 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:52.384 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:52.385 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-02-09T22:12:52.386 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:52.386 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 964 k 2024-02-09T22:12:52.386 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-02-09T22:12:52.386 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:52.512 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 627 kB/s | 79 kB 00:00 2024-02-09T22:12:52.713 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-02-09T22:12:52.714 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:52.715 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.9 MB/s | 964 kB 00:00 2024-02-09T22:12:52.715 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:52.736 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:52.736 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:52.812 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:52.815 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:52.993 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:53.130 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-09T22:12:53.204 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-09T22:12:53.946 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-09T22:12:53.946 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-09T22:12:54.263 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-09T22:12:54.263 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:54.263 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:54.264 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-09T22:12:54.264 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:54.264 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:54.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-02-09T22:12:55.186 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:55 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:55.345 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-02-09T22:12:55.346 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:55.347 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 332 k 2024-02-09T22:12:55.347 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-02-09T22:12:55.347 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:55.468 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 568 kB/s | 69 kB 00:00 2024-02-09T22:12:55.635 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 916 kB/s | 264 kB 00:00 2024-02-09T22:12:55.636 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:55.636 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-02-09T22:12:55.636 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:55.645 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:55.645 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:55.684 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:55.684 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:56.018 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-09T22:12:56.062 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-09T22:12:56.077 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-09T22:12:56.078 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:12:56.078 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-09T22:12:56.078 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:56.813 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-09T22:12:57.165 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-09T22:12:57.166 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:57.166 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:57.166 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:12:57.166 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-09T22:12:57.166 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:57.166 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:12:57.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-02-09T22:12:58.126 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:58 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:12:58.288 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:12:58.289 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:58.290 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-02-09T22:12:58.290 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-02-09T22:12:58.290 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:12:58.756 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 271 kB/s | 127 kB 00:00 2024-02-09T22:12:58.757 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:12:58.757 INFO:teuthology.orchestra.run.smithi134.stdout:Total 271 kB/s | 127 kB 00:00 2024-02-09T22:12:58.757 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:12:58.763 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:12:58.763 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:12:58.818 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:12:58.819 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:12:58.964 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:12:59.063 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:12:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:12:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:12:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:59.874 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:12:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:12:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:12:59.874 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:00.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-02-09T22:13:00.731 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:00 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:00.839 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:00.891 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:00.892 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:00.892 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:00.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-02-09T22:13:01.554 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:01 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:01.715 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:01.716 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:01.716 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-02-09T22:13:01.718 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 140 k 2024-02-09T22:13:01.718 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:13:01.925 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 150 kB/s | 31 kB 00:00 2024-02-09T22:13:01.926 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:13:01.926 INFO:teuthology.orchestra.run.smithi134.stdout:Total 149 kB/s | 31 kB 00:00 2024-02-09T22:13:01.926 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:13:01.930 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:13:01.930 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:13:01.953 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:13:01.953 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:13:02.095 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:13:02.199 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:02.606 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:02.895 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:02.895 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:02.895 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:13:02.895 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:13:02.895 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:02.895 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:03.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-02-09T22:13:03.682 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:03 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:03.789 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:03.840 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:03.841 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:03.841 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:03.886 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-02-09T22:13:04.473 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:04 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:04.579 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:04.630 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:04.631 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:04.631 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:04.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-02-09T22:13:05.271 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:05 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:05.376 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:05.427 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:05.428 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:05.428 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:05.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-02-09T22:13:06.066 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:06 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:06.171 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:06.222 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:06.223 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:06.223 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:06.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-02-09T22:13:06.856 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:06 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:06.965 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:07.017 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:07.018 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:07.018 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:07.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-02-09T22:13:07.676 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:07 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:07.785 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-09T22:13:07.836 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:07.837 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-09T22:13:07.837 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:07.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-02-09T22:13:08.485 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:08 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:08.644 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:08.644 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-02-09T22:13:08.645 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:13:09.161 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 165 kB/s | 85 kB 00:00 2024-02-09T22:13:09.161 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:13:09.162 INFO:teuthology.orchestra.run.smithi134.stdout:Total 165 kB/s | 85 kB 00:00 2024-02-09T22:13:09.162 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:13:09.169 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:13:09.169 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:13:09.225 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:13:09.225 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:13:09.361 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:13:09.468 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:09.956 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:10.255 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:10.255 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:10.255 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:13:10.255 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:13:10.255 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:10.255 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:10.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-02-09T22:13:11.121 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:11 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:11.281 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:11.281 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-02-09T22:13:11.282 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:13:14.080 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-02-09T22:13:14.080 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:13:14.081 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-02-09T22:13:14.081 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:13:14.092 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:13:14.092 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:13:14.169 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:13:14.169 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:13:14.588 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:13:14.639 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:14.663 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:14.663 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:13:14.663 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-09T22:13:14.663 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:13:14.663 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:13:14.663 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:15.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:15.890 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:15.890 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:13:15.890 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:13:15.890 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:15.890 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:16.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-02-09T22:13:16.791 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:16 ago on Fri 09 Feb 2024 10:11:00 PM UTC. 2024-02-09T22:13:16.947 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 171 k 2024-02-09T22:13:16.948 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-02-09T22:13:16.949 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-09T22:13:17.449 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 341 kB/s | 171 kB 00:00 2024-02-09T22:13:17.450 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:13:17.450 INFO:teuthology.orchestra.run.smithi134.stdout:Total 340 kB/s | 171 kB 00:00 2024-02-09T22:13:17.450 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-09T22:13:17.458 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-09T22:13:17.458 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-09T22:13:17.513 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-09T22:13:17.514 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-09T22:13:17.661 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-09T22:13:17.782 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:18.285 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:18.603 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-09T22:13:18.604 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:18.604 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-09T22:13:18.604 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:13:18.604 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-09T22:13:18.604 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-09T22:13:18.862 DEBUG:teuthology.parallel:result is None 2024-02-09T22:13:18.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:13:19.021 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T22:13:19.042 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-1361.g290212a1.el9 2024-02-09T22:13:19.043 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-09T22:13:19.043 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-09T22:13:19.045 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-09T22:13:19.045 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:13:19.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T22:13:19.111 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-09T22:13:19.111 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:13:19.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T22:13:19.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T22:13:19.245 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-09T22:13:19.245 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:13:19.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T22:13:19.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T22:13:19.380 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-09T22:13:19.380 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:13:19.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T22:13:19.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T22:13:19.527 INFO:teuthology.run_tasks:Running task workunit... 2024-02-09T22:13:19.541 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:13:19.541 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-09T22:13:19.541 INFO:tasks.workunit:timeout=3h 2024-02-09T22:13:19.541 INFO:tasks.workunit:cleanup=True 2024-02-09T22:13:19.542 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-09T22:13:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:13:19.582 INFO:teuthology.orchestra.run.smithi134.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-09T22:13:19.582 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-09T22:13:19.637 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-09T22:13:19.637 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-09T22:13:19.692 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:13:19.747 INFO:tasks.workunit.client.0.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-09T22:14:03.235 INFO:tasks.workunit.client.0.smithi134.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.236 INFO:tasks.workunit.client.0.smithi134.stderr: git switch -c 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr:Or undo this operation with: 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr: git switch - 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-02-09T22:14:03.237 INFO:tasks.workunit.client.0.smithi134.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-09T22:14:03.239 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-09T22:14:03.294 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-09T22:14:03.294 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-09T22:14:03.349 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-02-09T22:14:03.350 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-02-09T22:14:03.350 DEBUG:teuthology.orchestra.run.smithi134:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-02-09T22:14:03.417 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-09T22:14:03.423 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-09T22:14:03.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-09T22:14:03.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-09T22:14:03.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-09T22:14:03.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-02-09T22:14:03.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-02-09T22:14:03.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none ' 2024-02-09T22:14:03.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-02-09T22:14:03.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-09T22:14:03.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-02-09T22:14:03.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-09T22:14:03.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:14:03.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:14:03.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:14:03.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:14:03.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:14:03.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:14:03.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:14:03.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:14:03.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:14:03.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:14:03.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:14:03.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:14:03.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:14:03.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:14:03.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:14:03.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:14:03.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:14:03.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-09T22:14:03.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-09T22:14:03.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:14:03.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:14:03.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:14:03.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28244 2024-02-09T22:14:03.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:14:03.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:14:03.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.28244 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-09T22:14:03.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-02-09T22:14:03.446 INFO:tasks.workunit.client.0.smithi134.stdout:Dir: td/osd-mapper 2024-02-09T22:14:03.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-09T22:14:03.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-09T22:14:05.117 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 4012134b-51ff-46be-82f1-7142aa0c4fa2 2024-02-09T22:14:05.351 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:14:05.433 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.430+0000 7fed63c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:05.435 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.432+0000 7fed63c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:05.436 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.433+0000 7fed63c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:05.437 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.433+0000 7fed63c22640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-09T22:14:05.437 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.433+0000 7fed63c22640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-09T22:14:05.437 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.433+0000 7fed63c22640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-09T22:14:05.519 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:05.515+0000 7fed63c22640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-02-09T22:14:10.739 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-02-09T22:14:11.108 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-02-09T22:14:11.137 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:11.133+0000 7f757c811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:11.138 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:11.135+0000 7f757c811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:11.136+0000 7f757c811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:11.376 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:14:11.667 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:11.664+0000 7f757c811640 -1 Falling back to public interface 2024-02-09T22:14:12.649 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:14:13.929 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:14:13.944 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:13.941+0000 7f757c811640 -1 osd.0 0 log_to_monitors true 2024-02-09T22:14:15.231 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:14:16.532 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-02-09T22:14:17.811 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-02-09T22:14:19.089 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-02-09T22:14:20.370 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-02-09T22:14:21.714 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-02-09T22:14:22.994 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-02-09T22:14:24.275 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-02-09T22:14:25.674 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-02-09T22:14:27.480 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-02-09T22:14:29.072 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-02-09T22:14:30.514 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-02-09T22:14:31.940 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-02-09T22:14:33.558 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-02-09T22:14:34.945 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-02-09T22:14:35.307 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1505471990,v1:127.0.0.1:6803/1505471990] [v2:127.0.0.1:6804/1505471990,v1:127.0.0.1:6805/1505471990] exists,up 4012134b-51ff-46be-82f1-7142aa0c4fa2 2024-02-09T22:14:35.310 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 158b15f5-7f79-4654-a5f8-09a572b58603 2024-02-09T22:14:35.687 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:14:35.728 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.725+0000 7f26ad3ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:35.771 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.768+0000 7f26ad3ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.769+0000 7f26ad3ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.769+0000 7f26ad3ff640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-09T22:14:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.769+0000 7f26ad3ff640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-09T22:14:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.769+0000 7f26ad3ff640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-09T22:14:35.847 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:35.843+0000 7f26ad3ff640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-02-09T22:14:43.624 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-02-09T22:14:43.973 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-02-09T22:14:44.001 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:43.997+0000 7f13c8319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:44.002 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:43.999+0000 7f13c8319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:44.003 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:44.000+0000 7f13c8319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:14:44.250 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:14:44.525 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:44.522+0000 7f13c8319640 -1 Falling back to public interface 2024-02-09T22:14:45.518 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:14:46.795 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:14:46.882 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:14:46.878+0000 7f13c8319640 -1 osd.1 0 log_to_monitors true 2024-02-09T22:14:48.090 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:14:49.371 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-02-09T22:14:50.693 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-02-09T22:14:51.975 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-02-09T22:14:53.254 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-02-09T22:14:54.645 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-02-09T22:14:55.924 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-02-09T22:14:57.203 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-02-09T22:14:58.586 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-02-09T22:15:00.358 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-02-09T22:15:02.436 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-02-09T22:15:03.845 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-02-09T22:15:05.430 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-02-09T22:15:07.046 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-02-09T22:15:08.326 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-02-09T22:15:09.636 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-02-09T22:15:09.907 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3768791452,v1:127.0.0.1:6811/3768791452] [v2:127.0.0.1:6812/3768791452,v1:127.0.0.1:6813/3768791452] exists,up 158b15f5-7f79-4654-a5f8-09a572b58603 2024-02-09T22:15:09.911 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 f35bfb50-a106-4d3b-aca2-e1fda46ca1c7 2024-02-09T22:15:10.297 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:15:10.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.335+0000 7f5d42011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:15:10.385 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.381+0000 7f5d42011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:15:10.385 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.382+0000 7f5d42011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:15:10.386 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.382+0000 7f5d42011640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-09T22:15:10.386 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.382+0000 7f5d42011640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-09T22:15:10.386 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.382+0000 7f5d42011640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-09T22:15:10.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:10.444+0000 7f5d42011640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-02-09T22:15:16.547 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-02-09T22:15:16.896 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-02-09T22:15:16.923 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:16.920+0000 7f8b30a99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:15:16.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:16.922+0000 7f8b30a99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:15:16.926 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:16.923+0000 7f8b30a99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:15:17.166 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:15:17.443 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:17.440+0000 7f8b30a99640 -1 Falling back to public interface 2024-02-09T22:15:18.440 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:15:19.711 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:15:19.771 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:15:19.767+0000 7f8b30a99640 -1 osd.2 0 log_to_monitors true 2024-02-09T22:15:21.005 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:15:22.280 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-02-09T22:15:23.658 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-02-09T22:15:24.937 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-02-09T22:15:26.261 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-02-09T22:15:27.667 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-02-09T22:15:28.987 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-02-09T22:15:30.270 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-02-09T22:15:31.557 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-02-09T22:15:33.046 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-02-09T22:15:34.882 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-02-09T22:15:36.328 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-02-09T22:15:38.213 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-02-09T22:15:39.681 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-02-09T22:15:41.409 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-02-09T22:15:43.104 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-02-09T22:15:43.385 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1161253680,v1:127.0.0.1:6819/1161253680] [v2:127.0.0.1:6820/1161253680,v1:127.0.0.1:6821/1161253680] exists,up f35bfb50-a106-4d3b-aca2-e1fda46ca1c7 2024-02-09T22:15:43.848 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' created 2024-02-09T22:15:45.790 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836499 2024-02-09T22:15:47.470 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672971 2024-02-09T22:15:47.741 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509447 2024-02-09T22:15:55.212 INFO:tasks.workunit.client.0.smithi134.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-02-09T22:15:55.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-02-09T22:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: { 2024-02-09T22:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-09T22:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: { 2024-02-09T22:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-09T22:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: { 2024-02-09T22:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-09T22:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:55.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-02-09T22:15:55.468 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: { 2024-02-09T22:15:55.468 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-09T22:15:55.468 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:55.482 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: { 2024-02-09T22:15:55.483 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-09T22:15:55.483 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:55.496 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: { 2024-02-09T22:15:55.496 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-09T22:15:55.496 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-09T22:15:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-09T22:15:55.787 INFO:tasks.workunit.client.0.smithi134.stdout:created pool test snap snap01 2024-02-09T22:15:55.995 INFO:tasks.workunit.client.0.smithi134.stdout:created pool test snap snap02 2024-02-09T22:15:55.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-02-09T22:15:55.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-09T22:15:55.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-09T22:15:56.433 INFO:tasks.workunit.client.0.smithi134.stdout:created pool test snap snap13 2024-02-09T22:15:56.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-02-09T22:15:56.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-09T22:15:56.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-09T22:15:56.785 INFO:tasks.workunit.client.0.smithi134.stdout:created pool test snap snap24 2024-02-09T22:15:57.001 INFO:tasks.workunit.client.0.smithi134.stdout:created pool test snap snap25 2024-02-09T22:15:57.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 18142 2024-02-09T22:15:57.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-02-09T22:15:57.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-09T22:15:57.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-02-09T22:15:57.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-02-09T22:15:57.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-09T22:15:57.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-02-09T22:15:57.869 INFO:tasks.workunit.client.0.smithi134.stdout:pgid is 1.3 (primary: osd.1) 2024-02-09T22:15:57.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-02-09T22:15:57.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-02-09T22:15:57.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-02-09T22:15:57.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-02-09T22:15:57.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-02-09T22:15:57.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-02-09T22:15:58.123 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs_brief 2024-02-09T22:15:58.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-02-09T22:15:58.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-02-09T22:15:58.134 INFO:tasks.workunit.client.0.smithi134.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-02-09T22:15:58.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-02-09T22:15:58.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-09T22:15:58.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-09T22:15:58.136 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-09T22:15:58.136 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:15:58.136 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:15:58.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.28244/ceph-osd.1.asok 2024-02-09T22:15:58.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-02-09T22:15:58.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.28244/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-02-09T22:15:58.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:15:58.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout:{"success":true}{ 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "objxxx", 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "clones": [ 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [ 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap01" 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.254 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap02" 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 6, 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "overlaps": [] 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [ 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap13" 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-02-09T22:15:58.255 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 6, 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "overlaps": [] 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 5, 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [ 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 4, 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap24" 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 5, 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap25" 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:15:58.256 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 5, 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: "overlaps": [] 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: "id": "head", 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [], 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 6 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:58.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-09T22:15:58.349 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:15:58.349 INFO:tasks.workunit.client.0.smithi134.stdout: "deep": true, 2024-02-09T22:15:58.349 INFO:tasks.workunit.client.0.smithi134.stdout: "must": true, 2024-02-09T22:15:58.349 INFO:tasks.workunit.client.0.smithi134.stdout: "stamp": "0.000000" 2024-02-09T22:15:58.349 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:15:58.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-02-09T22:16:01.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-09T22:16:01.622 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:16:01.622 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-09T22:15:59.059665+0000 28'7 28:47 [1,2,0] 1 [1,2,0] 1 28'7 2024-02-09T22:15:59.059582+0000 28'7 2024-02-09T22:15:59.059582+0000 0 1 periodic scrub scheduled @ 2024-02-10T22:15:59.059582+0000 4 0 2024-02-09T22:16:01.622 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:15:45.643514+0000 0'0 23:19 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:01.622 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:15:44.702588+0000 0'0 28:33 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:01.623 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:15:45.263959+0000 0'0 28:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:01.623 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:01.623 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:16:01.623 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:16:01.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-02-09T22:16:01.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-09T22:16:01.884 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:16:01.884 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-09T22:15:59.059665+0000 28'7 28:47 [1,2,0] 1 [1,2,0] 1 28'7 2024-02-09T22:15:59.059582+0000 28'7 2024-02-09T22:15:59.059582+0000 0 1 periodic scrub scheduled @ 2024-02-10T22:15:59.059582+0000 4 0 2024-02-09T22:16:01.884 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:15:45.643514+0000 0'0 23:19 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:01.884 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:15:44.702588+0000 0'0 28:33 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:01.885 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:15:45.263959+0000 0'0 28:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:01.885 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:01.885 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:16:01.885 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:16:01.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-02-09T22:16:02.308 INFO:tasks.workunit.client.0.smithi134.stderr:noscrub is set 2024-02-09T22:16:02.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-02-09T22:16:02.827 INFO:tasks.workunit.client.0.smithi134.stderr:nodeep-scrub is set 2024-02-09T22:16:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-02-09T22:16:07.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-02-09T22:16:07.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-02-09T22:16:07.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:16:07.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:16:07.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:16:07.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:16:07.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:16:08.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:16:08.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:08.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-09T22:16:08.175 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bdev(0x564afdb8e000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 0 bdev(0x564afdb8e000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bdev(0x564afdb8e000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bdev(0x564afdb8e700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 0 bdev(0x564afdb8e700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bdev(0x564afdb8e700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-09T22:16:08.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.172+0000 7f092002d980 1 bluefs mount 2024-02-09T22:16:08.186 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.182+0000 7f092002d980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:08.219 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.216+0000 7f092002d980 1 bluefs mount shared_bdev_used = 0 2024-02-09T22:16:08.219 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.216+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:08.222 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:08.222 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.218+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.219+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.219+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.219+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.219+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 freelist init 2024-02-09T22:16:08.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.251+0000 7f092002d980 1 freelist _read_cfg 2024-02-09T22:16:08.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.252+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 8.39236e-07 2024-02-09T22:16:08.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.252+0000 7f092002d980 1 bluefs umount 2024-02-09T22:16:08.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.252+0000 7f092002d980 1 bdev(0x564afdb8e700 td/osd-mapper/0/block) close 2024-02-09T22:16:08.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.434+0000 7f092002d980 1 bdev(0x564afdb8e700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-09T22:16:08.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.434+0000 7f092002d980 0 bdev(0x564afdb8e700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-09T22:16:08.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.434+0000 7f092002d980 1 bdev(0x564afdb8e700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:08.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.434+0000 7f092002d980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-09T22:16:08.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.434+0000 7f092002d980 1 bluefs mount 2024-02-09T22:16:08.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.435+0000 7f092002d980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:08.444 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.441+0000 7f092002d980 1 bluefs mount shared_bdev_used = 27459584 2024-02-09T22:16:08.444 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.441+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:08.447 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:08.447 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.443+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.444+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.444+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.444+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.444+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:08.444+0000 7f092002d980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:09.163 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.159+0000 7f092002d980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:09.164 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 0b 05 58 02 00 00 b8 db cc d1 fe 29 46 a0 bb c8 |..X........)F...| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 e6 15 85 33 22 dc 00 00 00 00 1e 00 00 00 00 00 |...3"...........| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:* 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-09T22:16:09.165 INFO:tasks.workunit.client.0.smithi134.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-09T22:16:09.166 INFO:tasks.workunit.client.0.smithi134.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 40 12 13 4b 51 ff |re........@..KQ.| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:00000230 46 be 82 f1 71 42 aa 0c 4f a2 00 00 00 00 00 00 |F...qB..O.......| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:00000240 00 00 1e 00 00 00 ca a3 c6 65 8f 00 9b 28 01 00 |.........e...(..| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout:0000025e 2024-02-09T22:16:09.167 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-09T22:16:09.168 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:0000003f 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:09.169 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:09.170 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:09.171 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.160+0000 7f092002d980 1 bluefs umount 2024-02-09T22:16:09.172 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.160+0000 7f092002d980 1 bdev(0x564afdb8e700 td/osd-mapper/0/block) close 2024-02-09T22:16:09.408 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.405+0000 7f092002d980 1 freelist shutdown 2024-02-09T22:16:09.408 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.405+0000 7f092002d980 1 bdev(0x564afdb8e000 td/osd-mapper/0/block) close 2024-02-09T22:16:09.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:09.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-09T22:16:09.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-02-09T22:16:09.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.462+0000 7f9228c35980 1 bdev(0x562cec716000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-09T22:16:09.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.462+0000 7f9228c35980 0 bdev(0x562cec716000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-09T22:16:09.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 1 bdev(0x562cec716000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:09.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-09T22:16:09.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 1 bdev(0x562cec716700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-09T22:16:09.467 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 0 bdev(0x562cec716700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-09T22:16:09.467 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 1 bdev(0x562cec716700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:09.467 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-09T22:16:09.467 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.463+0000 7f9228c35980 1 bluefs mount 2024-02-09T22:16:09.471 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.468+0000 7f9228c35980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:09.480 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.477+0000 7f9228c35980 1 bluefs mount shared_bdev_used = 0 2024-02-09T22:16:09.480 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.477+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:09.483 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:09.483 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:09.483 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:09.483 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.479+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.480+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.480+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.511+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.511+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-02-09T22:16:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.511+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-02-09T22:16:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.511+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-09T22:16:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.511+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-02-09T22:16:09.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.512+0000 7f9228c35980 1 freelist init 2024-02-09T22:16:09.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.512+0000 7f9228c35980 1 freelist _read_cfg 2024-02-09T22:16:09.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.512+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.72206e-07 2024-02-09T22:16:09.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.513+0000 7f9228c35980 1 bluefs umount 2024-02-09T22:16:09.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.513+0000 7f9228c35980 1 bdev(0x562cec716700 td/osd-mapper/2/block) close 2024-02-09T22:16:09.731 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.725+0000 7f9228c35980 1 bdev(0x562cec716700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-09T22:16:09.731 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.725+0000 7f9228c35980 0 bdev(0x562cec716700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-09T22:16:09.732 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.725+0000 7f9228c35980 1 bdev(0x562cec716700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:09.732 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.725+0000 7f9228c35980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-09T22:16:09.732 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.725+0000 7f9228c35980 1 bluefs mount 2024-02-09T22:16:09.732 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.726+0000 7f9228c35980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:09.735 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.732+0000 7f9228c35980 1 bluefs mount shared_bdev_used = 27459584 2024-02-09T22:16:09.735 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.732+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:09.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:09.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:09.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:09.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:09.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.734+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.735+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.735+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.735+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:09.735+0000 7f9228c35980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:10.476 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:10.472+0000 7f9228c35980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:10.476 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:10.473+0000 7f9228c35980 1 bluefs umount 2024-02-09T22:16:10.477 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:10.473+0000 7f9228c35980 1 bdev(0x562cec716700 td/osd-mapper/2/block) close 2024-02-09T22:16:10.707 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:10.704+0000 7f9228c35980 1 freelist shutdown 2024-02-09T22:16:10.707 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:10.704+0000 7f9228c35980 1 bdev(0x562cec716000 td/osd-mapper/2/block) close 2024-02-09T22:16:10.737 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:10.737 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:10.738 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:10.738 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:10.738 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:10.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:10.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-02-09T22:16:10.739 INFO:tasks.workunit.client.0.smithi134.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-02-09T22:16:10.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:10.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-09T22:16:11.396 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-09T22:16:11.396 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 0b 05 58 02 00 00 b8 db cc d1 fe 29 46 a0 bb c8 |..X........)F...| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 e6 15 85 33 22 dc 02 00 00 00 1e 00 00 00 00 00 |...3"...........| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:* 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-09T22:16:11.397 INFO:tasks.workunit.client.0.smithi134.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-09T22:16:11.398 INFO:tasks.workunit.client.0.smithi134.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 f3 5b fb 50 a1 06 |re.........[.P..| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:00000230 4d 3b ac a2 e1 fd a4 6c a1 c7 00 00 00 00 00 00 |M;.....l........| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:00000240 00 00 1e 00 00 00 0d a4 c6 65 73 62 40 35 01 00 |.........esb@5..| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:0000025e 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.399 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout:0000003f 2024-02-09T22:16:11.400 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:11.401 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:11.402 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:11.403 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.404 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-09T22:16:11.404 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.404 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-09T22:16:11.404 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:11.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-09T22:16:11.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-02-09T22:16:11.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-09T22:16:11.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-02-09T22:16:13.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-02-09T22:16:13.323 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-02-09T22:16:13.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-02-09T22:16:13.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:13.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-02-09T22:16:13.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-02-09T22:16:13.326 INFO:tasks.workunit.client.0.smithi134.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-02-09T22:16:13.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-09T22:16:13.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-09T22:16:13.350 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.346+0000 7f8dab35b980 1 bdev(0x560bfe896000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.346+0000 7f8dab35b980 0 bdev(0x560bfe896000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.346+0000 7f8dab35b980 1 bdev(0x560bfe896000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.347+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.347+0000 7f8dab35b980 1 bdev(0x560bfe896700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.347+0000 7f8dab35b980 0 bdev(0x560bfe896700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.347+0000 7f8dab35b980 1 bdev(0x560bfe896700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.347+0000 7f8dab35b980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-09T22:16:13.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.347+0000 7f8dab35b980 1 bluefs mount 2024-02-09T22:16:13.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.356+0000 7f8dab35b980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:13.369 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.366+0000 7f8dab35b980 1 bluefs mount shared_bdev_used = 0 2024-02-09T22:16:13.370 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.366+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:13.372 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.368+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.369+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.369+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.369+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:13.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.369+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.373+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.373+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.373+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.373+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.373+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.374+0000 7f8dab35b980 1 freelist init 2024-02-09T22:16:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.374+0000 7f8dab35b980 1 freelist _read_cfg 2024-02-09T22:16:13.378 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.374+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 8.39236e-07 2024-02-09T22:16:13.378 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.374+0000 7f8dab35b980 1 bluefs umount 2024-02-09T22:16:13.378 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.374+0000 7f8dab35b980 1 bdev(0x560bfe896700 td/osd-mapper/0/block) close 2024-02-09T22:16:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.612+0000 7f8dab35b980 1 bdev(0x560bfe896700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-09T22:16:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.612+0000 7f8dab35b980 0 bdev(0x560bfe896700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-09T22:16:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.612+0000 7f8dab35b980 1 bdev(0x560bfe896700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.612+0000 7f8dab35b980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-09T22:16:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.612+0000 7f8dab35b980 1 bluefs mount 2024-02-09T22:16:13.626 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.623+0000 7f8dab35b980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:13.635 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.631+0000 7f8dab35b980 1 bluefs mount shared_bdev_used = 9568256 2024-02-09T22:16:13.635 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.631+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:13.638 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:13.638 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:13.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.634+0000 7f8dab35b980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.888+0000 7f8dab35b980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:13.893 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-09T22:16:13.893 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 0b 05 58 02 00 00 b8 db cc d1 fe 29 46 a0 bb c8 |..X........)F...| 2024-02-09T22:16:13.893 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 e6 15 85 33 22 dc 00 00 00 00 1e 00 00 00 00 00 |...3"...........| 2024-02-09T22:16:13.893 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-09T22:16:13.893 INFO:tasks.workunit.client.0.smithi134.stdout:* 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-09T22:16:13.894 INFO:tasks.workunit.client.0.smithi134.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-09T22:16:13.895 INFO:tasks.workunit.client.0.smithi134.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 40 12 13 4b 51 ff |re........@..KQ.| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000230 46 be 82 f1 71 42 aa 0c 4f a2 00 00 00 00 00 00 |F...qB..O.......| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000240 00 00 1e 00 00 00 ca a3 c6 65 8f 00 9b 28 01 00 |.........e...(..| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:0000025e 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:13.896 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:0000003f 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:13.897 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:13.898 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:13.899 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:13.900 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.889+0000 7f8dab35b980 1 bluefs umount 2024-02-09T22:16:13.901 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:13.889+0000 7f8dab35b980 1 bdev(0x560bfe896700 td/osd-mapper/0/block) close 2024-02-09T22:16:14.128 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:14.125+0000 7f8dab35b980 1 freelist shutdown 2024-02-09T22:16:14.128 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:14.125+0000 7f8dab35b980 1 bdev(0x560bfe896000 td/osd-mapper/0/block) close 2024-02-09T22:16:14.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-09T22:16:14.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-09T22:16:14.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-09T22:16:15.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:15.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:15.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-09T22:16:15.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-09T22:16:15.413 INFO:tasks.workunit.client.0.smithi134.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:15.414 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-09T22:16:15.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.QlTzu6e2Kh_the_val 2024-02-09T22:16:15.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:15.415 INFO:tasks.workunit.client.0.smithi134.stdout:Value dumped in: /tmp/tmp.QlTzu6e2Kh_the_val 2024-02-09T22:16:15.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.QlTzu6e2Kh_the_val' 2024-02-09T22:16:15.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.QlTzu6e2Kh_the_val 2024-02-09T22:16:15.952 INFO:tasks.workunit.client.0.smithi134.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-09T22:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.QlTzu6e2Kh_the_val 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-09T22:16:16.229 INFO:tasks.workunit.client.0.smithi134.stdout:0000073 2024-02-09T22:16:16.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-09T22:16:16.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:17.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.QlTzu6e2Kh_the_val 2024-02-09T22:16:18.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:18.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:18.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-02-09T22:16:18.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-02-09T22:16:18.067 INFO:tasks.workunit.client.0.smithi134.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-02-09T22:16:18.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-09T22:16:18.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-09T22:16:18.091 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bdev(0x55939bd36000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-09T22:16:18.091 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 0 bdev(0x55939bd36000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-02-09T22:16:18.091 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bdev(0x55939bd36000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:18.091 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-09T22:16:18.091 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bdev(0x55939bd36700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-09T22:16:18.092 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 0 bdev(0x55939bd36700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-02-09T22:16:18.092 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bdev(0x55939bd36700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:18.092 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-02-09T22:16:18.092 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.087+0000 7fd203439980 1 bluefs mount 2024-02-09T22:16:18.103 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.099+0000 7fd203439980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:18.112 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.109+0000 7fd203439980 1 bluefs mount shared_bdev_used = 0 2024-02-09T22:16:18.112 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.109+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:18.116 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:18.116 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.111+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.112+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.112+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.112+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.112+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:18.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.112+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:18.123 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 freelist init 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 freelist _read_cfg 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.117+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 6.10353e-07 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.118+0000 7fd203439980 1 bluefs umount 2024-02-09T22:16:18.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.118+0000 7fd203439980 1 bdev(0x55939bd36700 td/osd-mapper/1/block) close 2024-02-09T22:16:18.374 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.371+0000 7fd203439980 1 bdev(0x55939bd36700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-09T22:16:18.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.371+0000 7fd203439980 0 bdev(0x55939bd36700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-02-09T22:16:18.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.371+0000 7fd203439980 1 bdev(0x55939bd36700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:18.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.371+0000 7fd203439980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-02-09T22:16:18.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.371+0000 7fd203439980 1 bluefs mount 2024-02-09T22:16:18.378 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.374+0000 7fd203439980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:18.394 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.391+0000 7fd203439980 1 bluefs mount shared_bdev_used = 9568256 2024-02-09T22:16:18.395 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.391+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.393+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.393+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:18.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.394+0000 7fd203439980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:18.732 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.728+0000 7fd203439980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 0b 05 58 02 00 00 b8 db cc d1 fe 29 46 a0 bb c8 |..X........)F...| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 e6 15 85 33 22 dc 01 00 00 00 1e 00 00 00 00 00 |...3"...........| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:* 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-09T22:16:18.734 INFO:tasks.workunit.client.0.smithi134.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-09T22:16:18.735 INFO:tasks.workunit.client.0.smithi134.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000220 72 65 1e 00 00 00 0a 00 00 00 15 8b 15 f5 7f 79 |re.............y| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000230 46 54 a5 f8 09 a5 72 b5 86 03 00 00 00 00 00 00 |FT....r.........| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000240 00 00 1e 00 00 00 ec a3 c6 65 92 e0 a7 1b 01 00 |.........e......| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-02-09T22:16:18.736 INFO:tasks.workunit.client.0.smithi134.stdout:0000025e 2024-02-09T22:16:18.737 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-09T22:16:18.739 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:0000003f 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.740 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:18.741 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:18.742 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.743 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-09T22:16:18.744 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.744 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-09T22:16:18.744 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:18.744 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.729+0000 7fd203439980 1 bluefs umount 2024-02-09T22:16:18.744 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.729+0000 7fd203439980 1 bdev(0x55939bd36700 td/osd-mapper/1/block) close 2024-02-09T22:16:18.945 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.941+0000 7fd203439980 1 freelist shutdown 2024-02-09T22:16:18.945 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:18.941+0000 7fd203439980 1 bdev(0x55939bd36000 td/osd-mapper/1/block) close 2024-02-09T22:16:19.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-09T22:16:19.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-09T22:16:19.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-09T22:16:19.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:19.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:19.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-09T22:16:19.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-09T22:16:19.904 INFO:tasks.workunit.client.0.smithi134.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:19.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-09T22:16:19.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.DSkodPi5a9_the_val 2024-02-09T22:16:19.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:19.906 INFO:tasks.workunit.client.0.smithi134.stdout:Value dumped in: /tmp/tmp.DSkodPi5a9_the_val 2024-02-09T22:16:19.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.DSkodPi5a9_the_val' 2024-02-09T22:16:19.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.DSkodPi5a9_the_val 2024-02-09T22:16:20.434 INFO:tasks.workunit.client.0.smithi134.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-09T22:16:20.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:20.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.DSkodPi5a9_the_val 2024-02-09T22:16:20.700 INFO:tasks.workunit.client.0.smithi134.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-09T22:16:20.700 INFO:tasks.workunit.client.0.smithi134.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-09T22:16:20.700 INFO:tasks.workunit.client.0.smithi134.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stdout:0000073 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-09T22:16:20.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:21.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.DSkodPi5a9_the_val 2024-02-09T22:16:22.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:22.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:22.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-02-09T22:16:22.785 INFO:tasks.workunit.client.0.smithi134.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-02-09T22:16:22.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-02-09T22:16:22.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-09T22:16:22.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-09T22:16:22.812 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.807+0000 7f9599d62980 1 bdev(0x55faafed8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-09T22:16:22.812 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.807+0000 7f9599d62980 0 bdev(0x55faafed8000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 1 bdev(0x55faafed8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 1 bdev(0x55faafed8700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 0 bdev(0x55faafed8700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 1 bdev(0x55faafed8700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-09T22:16:22.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.808+0000 7f9599d62980 1 bluefs mount 2024-02-09T22:16:22.828 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.824+0000 7f9599d62980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:22.837 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.834+0000 7f9599d62980 1 bluefs mount shared_bdev_used = 0 2024-02-09T22:16:22.837 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.834+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:22.840 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:22.840 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.836+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.837+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.837+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.837+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.841+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.841+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.841+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.841+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.841+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.842+0000 7f9599d62980 1 freelist init 2024-02-09T22:16:22.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.842+0000 7f9599d62980 1 freelist _read_cfg 2024-02-09T22:16:22.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.842+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.72206e-07 2024-02-09T22:16:22.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.842+0000 7f9599d62980 1 bluefs umount 2024-02-09T22:16:22.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:22.842+0000 7f9599d62980 1 bdev(0x55faafed8700 td/osd-mapper/2/block) close 2024-02-09T22:16:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.070+0000 7f9599d62980 1 bdev(0x55faafed8700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-09T22:16:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.070+0000 7f9599d62980 0 bdev(0x55faafed8700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-09T22:16:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.070+0000 7f9599d62980 1 bdev(0x55faafed8700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-09T22:16:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.070+0000 7f9599d62980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-09T22:16:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.070+0000 7f9599d62980 1 bluefs mount 2024-02-09T22:16:23.078 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.074+0000 7f9599d62980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-09T22:16:23.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.083+0000 7f9599d62980 1 bluefs mount shared_bdev_used = 9568256 2024-02-09T22:16:23.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.083+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-09T22:16:23.089 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-09T22:16:23.089 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.085+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.086+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-09T22:16:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.086+0000 7f9599d62980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-09T22:16:23.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.395+0000 7f9599d62980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-09T22:16:23.400 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-09T22:16:23.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 0b 05 58 02 00 00 b8 db cc d1 fe 29 46 a0 bb c8 |..X........)F...| 2024-02-09T22:16:23.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 e6 15 85 33 22 dc 02 00 00 00 1e 00 00 00 00 00 |...3"...........| 2024-02-09T22:16:23.400 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-09T22:16:23.400 INFO:tasks.workunit.client.0.smithi134.stdout:* 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-09T22:16:23.401 INFO:tasks.workunit.client.0.smithi134.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-09T22:16:23.402 INFO:tasks.workunit.client.0.smithi134.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 f3 5b fb 50 a1 06 |re.........[.P..| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000230 4d 3b ac a2 e1 fd a4 6c a1 c7 00 00 00 00 00 00 |M;.....l........| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000240 00 00 1e 00 00 00 0d a4 c6 65 73 62 40 35 01 00 |.........esb@5..| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:0000025e 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.403 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:23.404 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:23.404 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:23.404 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:23.404 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-09T22:16:23.404 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:23.404 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:0000003f 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-09T22:16:23.405 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000047 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:23.407 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:23.408 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:0000003b 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.409 INFO:tasks.workunit.client.0.smithi134.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-09T22:16:23.410 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:23.410 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.396+0000 7f9599d62980 1 bluefs umount 2024-02-09T22:16:23.410 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.396+0000 7f9599d62980 1 bdev(0x55faafed8700 td/osd-mapper/2/block) close 2024-02-09T22:16:23.613 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.610+0000 7f9599d62980 1 freelist shutdown 2024-02-09T22:16:23.614 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:23.610+0000 7f9599d62980 1 bdev(0x55faafed8000 td/osd-mapper/2/block) close 2024-02-09T22:16:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-09T22:16:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-09T22:16:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-09T22:16:24.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:24.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:24.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-09T22:16:24.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-09T22:16:24.621 INFO:tasks.workunit.client.0.smithi134.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:24.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-09T22:16:24.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.C2ACtru6k4_the_val 2024-02-09T22:16:24.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:24.623 INFO:tasks.workunit.client.0.smithi134.stdout:Value dumped in: /tmp/tmp.C2ACtru6k4_the_val 2024-02-09T22:16:24.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.C2ACtru6k4_the_val' 2024-02-09T22:16:24.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.C2ACtru6k4_the_val 2024-02-09T22:16:25.116 INFO:tasks.workunit.client.0.smithi134.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-09T22:16:25.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:25.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.C2ACtru6k4_the_val 2024-02-09T22:16:25.417 INFO:tasks.workunit.client.0.smithi134.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-09T22:16:25.417 INFO:tasks.workunit.client.0.smithi134.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-09T22:16:25.417 INFO:tasks.workunit.client.0.smithi134.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-09T22:16:25.417 INFO:tasks.workunit.client.0.smithi134.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-09T22:16:25.417 INFO:tasks.workunit.client.0.smithi134.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-09T22:16:25.417 INFO:tasks.workunit.client.0.smithi134.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-02-09T22:16:25.418 INFO:tasks.workunit.client.0.smithi134.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-09T22:16:25.418 INFO:tasks.workunit.client.0.smithi134.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-09T22:16:25.418 INFO:tasks.workunit.client.0.smithi134.stdout:0000073 2024-02-09T22:16:25.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-09T22:16:25.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:26.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.C2ACtru6k4_the_val 2024-02-09T22:16:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-02-09T22:16:27.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-02-09T22:16:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-02-09T22:16:27.257 INFO:tasks.workunit.client.0.smithi134.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-02-09T22:16:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-02-09T22:16:27.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-09T22:16:27.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:16:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28244/$cluster-$name.asok' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28244/$cluster-$name.asok' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:16:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:16:27.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:16:27.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-09T22:16:27.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-09T22:16:27.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-02-09T22:16:27.265 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-02-09T22:16:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-09T22:16:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28244/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-09T22:16:27.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-02-09T22:16:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-09T22:16:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-09T22:16:27.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-09T22:16:27.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-09T22:16:27.301 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:27.297+0000 7feaee606640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:27.303 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:27.299+0000 7feaee606640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:27.304 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:27.300+0000 7feaee606640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:27.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-09T22:16:27.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:16:27.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-09T22:16:27.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:16:27.550 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:16:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:16:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:16:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:16:27.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:27.815+0000 7feaee606640 -1 Falling back to public interface 2024-02-09T22:16:27.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:28.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:28.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:28.877 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:16:28.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:16:28.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:28.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:16:29.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:30.072 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:30.069+0000 7feaee606640 -1 osd.0 30 log_to_monitors true 2024-02-09T22:16:30.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:30.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:30.170 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:16:30.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:16:30.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:30.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:16:30.560 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 33 up_thru 18 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/2404353602,v1:127.0.0.1:6803/2404353602] [v2:127.0.0.1:6804/2404353602,v1:127.0.0.1:6805/2404353602] exists,up 4012134b-51ff-46be-82f1-7142aa0c4fa2 2024-02-09T22:16:30.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:16:30.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:16:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:16:30.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28244/$cluster-$name.asok' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28244/$cluster-$name.asok' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-09T22:16:30.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-09T22:16:30.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-02-09T22:16:30.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-09T22:16:30.568 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-02-09T22:16:30.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28244/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-09T22:16:30.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-02-09T22:16:30.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-09T22:16:30.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-09T22:16:30.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-09T22:16:30.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-09T22:16:30.599 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:30.596+0000 7fc06281c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:30.678 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:30.675+0000 7fc06281c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:30.679 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:30.676+0000 7fc06281c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:30.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-09T22:16:30.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:16:30.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-09T22:16:30.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:16:30.877 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:16:30.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:16:30.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:30.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:16:30.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:30.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:16:31.198 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:31.196+0000 7fc06281c640 -1 Falling back to public interface 2024-02-09T22:16:31.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:32.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:32.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:32.208 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:16:32.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:16:32.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:32.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:16:32.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:33.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:33.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:16:33.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:16:33.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:33.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:16:33.532 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:33.529+0000 7fc06281c640 -1 osd.1 30 log_to_monitors true 2024-02-09T22:16:33.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:34.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:34.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:34.832 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:16:34.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:16:34.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:34.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:16:35.115 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/76101333,v1:127.0.0.1:6811/76101333] [v2:127.0.0.1:6812/76101333,v1:127.0.0.1:6813/76101333] exists,up 158b15f5-7f79-4654-a5f8-09a572b58603 2024-02-09T22:16:35.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:16:35.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:16:35.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:16:35.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28244/$cluster-$name.asok' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28244/$cluster-$name.asok' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:16:35.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-09T22:16:35.123 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-02-09T22:16:35.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=b8dbccd1-fe29-46a0-bbc8-e615853322dc --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28244/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-09T22:16:35.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-02-09T22:16:35.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-09T22:16:35.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-09T22:16:35.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-09T22:16:35.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-09T22:16:35.153 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:35.149+0000 7f4391347640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:35.182 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:35.179+0000 7f4391347640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:35.183 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:35.180+0000 7f4391347640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:16:35.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-09T22:16:35.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:16:35.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-09T22:16:35.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:16:35.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:16:35.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:35.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:16:35.443 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:16:35.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:16:35.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:35.708+0000 7f4391347640 -1 Falling back to public interface 2024-02-09T22:16:35.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:36.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:36.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:36.728 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:16:36.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:16:36.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:36.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:16:37.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:37.931 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:16:37.927+0000 7f4391347640 -1 osd.2 30 log_to_monitors true 2024-02-09T22:16:38.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:38.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:38.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:16:38.021 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:16:38.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:38.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:16:38.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:16:39.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:16:39.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:16:39.314 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:16:39.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:16:39.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:16:39.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:16:39.674 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 40 up_thru 40 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/1710755560,v1:127.0.0.1:6819/1710755560] [v2:127.0.0.1:6820/1710755560,v1:127.0.0.1:6821/1710755560] exists,up f35bfb50-a106-4d3b-aca2-e1fda46ca1c7 2024-02-09T22:16:39.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:16:39.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:16:39.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:16:39.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-02-09T22:16:40.677 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-02-09T22:16:40.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-02-09T22:16:40.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:40.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-02-09T22:16:40.785 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:16:40.786 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-1361-g290212a1", 2024-02-09T22:16:40.786 INFO:tasks.workunit.client.0.smithi134.stdout: "release": "squid", 2024-02-09T22:16:40.786 INFO:tasks.workunit.client.0.smithi134.stdout: "release_type": "dev" 2024-02-09T22:16:40.786 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:16:40.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:40.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-02-09T22:16:40.891 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:16:40.891 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-1361-g290212a1", 2024-02-09T22:16:40.892 INFO:tasks.workunit.client.0.smithi134.stdout: "release": "squid", 2024-02-09T22:16:40.892 INFO:tasks.workunit.client.0.smithi134.stdout: "release_type": "dev" 2024-02-09T22:16:40.892 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:16:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-09T22:16:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-02-09T22:16:40.996 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:16:40.996 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-1361-g290212a1", 2024-02-09T22:16:40.996 INFO:tasks.workunit.client.0.smithi134.stdout: "release": "squid", 2024-02-09T22:16:40.996 INFO:tasks.workunit.client.0.smithi134.stdout: "release_type": "dev" 2024-02-09T22:16:40.996 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:16:41.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-02-09T22:16:41.032 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "objxxx", 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "clones": [ 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [ 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap01" 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.033 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap02" 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 6, 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "overlaps": [] 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [ 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap13" 2024-02-09T22:16:41.034 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 6, 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "overlaps": [] 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 5, 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [ 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 4, 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap24" 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 5, 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap25" 2024-02-09T22:16:41.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 5, 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: "overlaps": [] 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: "id": "head", 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: "snapshots": [], 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 6 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-02-09T22:16:41.036 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:16:41.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-02-09T22:16:41.480 INFO:tasks.workunit.client.0.smithi134.stderr:nodeep-scrub is unset 2024-02-09T22:16:41.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-02-09T22:16:41.982 INFO:tasks.workunit.client.0.smithi134.stderr:noscrub is unset 2024-02-09T22:16:41.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-09T22:16:41.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-02-09T22:16:42.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-02-09T22:16:42.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-09T22:16:42.540 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:16:42.540 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 peering 2024-02-09T22:16:38.519791+0000 28'7 39:57 [1,2,0] 1 [1,2,0] 1 28'7 2024-02-09T22:15:59.059582+0000 28'7 2024-02-09T22:15:59.059582+0000 0 1 no scrub is scheduled 4 0 2024-02-09T22:16:42.540 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:38.955886+0000 0'0 42:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:42.540 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-02-09T22:16:34.811728+0000 0'0 38:18 [0,1] 0 [0,1] 0 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:42.541 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-09T22:16:38.519768+0000 0'0 39:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 no scrub is scheduled 0 0 2024-02-09T22:16:42.541 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:42.541 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:16:42.541 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:16:42.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-02-09T22:16:44.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-09T22:16:44.657 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:16:44.657 INFO:tasks.workunit.client.0.smithi134.stdout: "deep": true, 2024-02-09T22:16:44.657 INFO:tasks.workunit.client.0.smithi134.stdout: "must": true, 2024-02-09T22:16:44.658 INFO:tasks.workunit.client.0.smithi134.stdout: "stamp": "0.000000" 2024-02-09T22:16:44.658 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:16:44.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-02-09T22:16:49.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-09T22:16:39.433926+0000 28'7 44:72 [1,2,0] 1 [1,2,0] 1 28'7 2024-02-09T22:15:59.059582+0000 28'7 2024-02-09T22:15:59.059582+0000 0 1 periodic scrub scheduled @ 2024-02-10T22:15:59.059582+0000 4 0 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:38.955886+0000 0'0 42:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:39.433751+0000 0'0 44:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:39.433709+0000 0'0 44:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:49.929 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:16:49.930 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:16:49.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:49.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-09T22:16:49.941 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:16:45.577+0000 7fc044941640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-02-09T22:16:49.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-02-09T22:16:49.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-09T22:16:49.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-02-09T22:16:49.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-02-09T22:16:49.946 INFO:tasks.workunit.client.0.smithi134.stdout:prev count: 1 2024-02-09T22:16:49.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-02-09T22:16:49.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-09T22:16:50.040 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:16:50.040 INFO:tasks.workunit.client.0.smithi134.stdout: "deep": true, 2024-02-09T22:16:50.040 INFO:tasks.workunit.client.0.smithi134.stdout: "must": true, 2024-02-09T22:16:50.040 INFO:tasks.workunit.client.0.smithi134.stdout: "stamp": "0.000000" 2024-02-09T22:16:50.040 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:16:50.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-02-09T22:16:55.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-09T22:16:55.314 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:16:55.314 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-09T22:16:50.665286+0000 28'7 44:90 [1,2,0] 1 [1,2,0] 1 28'7 2024-02-09T22:16:50.665203+0000 28'7 2024-02-09T22:16:50.665203+0000 0 1 periodic scrub scheduled @ 2024-02-10T22:16:50.665203+0000 4 0 2024-02-09T22:16:55.314 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:38.955886+0000 0'0 44:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:55.314 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:39.433751+0000 0'0 44:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:55.315 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:16:39.433709+0000 0'0 44:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-09T22:15:43.723664+0000 0'0 2024-02-09T22:15:43.723664+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:15:43.723664+0000 0 0 2024-02-09T22:16:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:16:55.315 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:16:55.315 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:16:55.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:55.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-09T22:16:55.326 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:16:45.577+0000 7fc044941640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-02-09T22:16:55.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-09T22:16:55.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-02-09T22:16:55.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-02-09T22:16:55.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-09T22:16:55.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-02-09T22:16:55.330 INFO:tasks.workunit.client.0.smithi134.stdout:current count: 1 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:16:55.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:16:55.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:16:55.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-02-09T22:16:55.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-09T22:16:55.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-09T22:16:55.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-02-09T22:16:55.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-09T22:16:57.173 INFO:tasks.workunit.client.0.smithi134.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:57.173 INFO:tasks.workunit.client.0.smithi134.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-09T22:16:57.173 INFO:tasks.workunit.client.0.smithi134.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-09T22:16:57.173 INFO:tasks.workunit.client.0.smithi134.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-09T22:16:57.174 INFO:tasks.workunit.client.0.smithi134.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:57.174 INFO:tasks.workunit.client.0.smithi134.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-09T22:16:57.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-09T22:16:57.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-02-09T22:16:57.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-09T22:16:57.175 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-02-09T22:16:57.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-02-09T22:16:57.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:16:57.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:16:58.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:16:58.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:16:58.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:16:58.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:16:58.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:16:58.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:16:58.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:16:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:16:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:16:58.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:16:58.085 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:16:58.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:16:58.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-09T22:16:58.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-09T22:16:58.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:16:58.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:16:58.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28244 2024-02-09T22:16:58.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:16:58.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:16:58.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:16:58.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:16:58.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:16:58.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:16:58.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:16:58.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:16:58.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:16:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:16:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:16:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:16:58.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:16:58.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:16:58.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-09T22:16:58.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-09T22:16:58.113 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:16:58.113 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.113 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28244 2024-02-09T22:16:58.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28244 2024-02-09T22:16:58.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:16:58.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:16:58.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-09T22:16:58.115 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-09T22:16:58.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-09T22:16:58.186 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-02-09T22:16:58.186 DEBUG:teuthology.orchestra.run.smithi134:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-02-09T22:16:58.248 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-02-09T22:16:58.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none ' 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-02-09T22:16:58.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_scrub_2' 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-02-09T22:16:58.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.44749 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:16:58.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:16:58.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:16:58.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:16:58.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:16:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:16:58.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-09T22:16:58.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-09T22:16:58.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:16:58.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:58.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44749 2024-02-09T22:16:58.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:16:58.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:16:58.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-02-09T22:16:58.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-09T22:16:58.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:58.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44749 2024-02-09T22:16:58.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-09T22:16:58.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-09T22:16:58.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-09T22:16:58.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-09T22:16:58.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-02-09T22:16:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:16:58.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-09T22:16:58.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-09T22:16:58.844 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44749/ceph-mon.a.asok 2024-02-09T22:16:58.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-09T22:16:58.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44749/ceph-mon.a.asok config get fsid 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44749/ceph-mon.a.asok 2024-02-09T22:16:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-09T22:16:58.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44749/ceph-mon.a.asok config get mon_host 2024-02-09T22:16:59.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-02-09T22:16:59.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-02-09T22:16:59.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-09T22:16:59.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-09T22:16:59.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-09T22:16:59.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-02-09T22:16:59.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-09T22:16:59.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-09T22:16:59.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:16:59.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:16:59.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:16:59.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:59.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:59.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:16:59.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-09T22:16:59.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-09T22:16:59.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-02-09T22:16:59.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-02-09T22:16:59.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-02-09T22:16:59.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-02-09T22:16:59.845 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-02-09T22:16:59.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-02-09T22:16:59.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:16:59.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-02-09T22:16:59.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:16:59.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:16:59.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:16:59.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:16:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ad06537-efd3-4e5d-b95e-f765ac1efbea 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8ad06537-efd3-4e5d-b95e-f765ac1efbea' 2024-02-09T22:16:59.854 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 8ad06537-efd3-4e5d-b95e-f765ac1efbea 2024-02-09T22:16:59.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:16:59.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbpMZllgSCMxAAOue0VWlmcdOiZqtsZ/HPDA== 2024-02-09T22:16:59.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbpMZllgSCMxAAOue0VWlmcdOiZqtsZ/HPDA=="}' 2024-02-09T22:16:59.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ad06537-efd3-4e5d-b95e-f765ac1efbea -i td/osd-recovery-scrub/0/new.json 2024-02-09T22:17:00.058 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:17:00.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-02-09T22:17:00.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBbpMZllgSCMxAAOue0VWlmcdOiZqtsZ/HPDA== --osd-uuid 8ad06537-efd3-4e5d-b95e-f765ac1efbea 2024-02-09T22:17:00.097 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.093+0000 7f1e1de11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:00.098 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.095+0000 7f1e1de11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:00.099 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.096+0000 7f1e1de11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:00.100 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.096+0000 7f1e1de11640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-09T22:17:00.100 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.096+0000 7f1e1de11640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-09T22:17:00.100 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.096+0000 7f1e1de11640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-09T22:17:00.167 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:00.164+0000 7f1e1de11640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-02-09T22:17:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-02-09T22:17:05.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:17:05.450 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-02-09T22:17:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-09T22:17:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:17:05.804 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-02-09T22:17:05.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-09T22:17:05.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-09T22:17:05.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:17:05.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:17:05.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:17:05.830 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:05.826+0000 7ff34be13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:05.865 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:05.861+0000 7ff34be13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:05.866 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:05.862+0000 7ff34be13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:06.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:17:06.077 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:17:06.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:06.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:17:06.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:06.401 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:06.398+0000 7ff34be13640 -1 Falling back to public interface 2024-02-09T22:17:07.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:07.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:07.348 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:17:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:17:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:17:07.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:08.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:08.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:08.631 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:17:08.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:17:08.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:08.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:17:08.682 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:08.679+0000 7ff34be13640 -1 osd.0 0 log_to_monitors true 2024-02-09T22:17:08.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:09.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:09.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:09.941 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:17:09.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:17:09.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:09.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:17:10.221 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3861344222,v1:127.0.0.1:6803/3861344222] [v2:127.0.0.1:6804/3861344222,v1:127.0.0.1:6805/3861344222] exists,up 8ad06537-efd3-4e5d-b95e-f765ac1efbea 2024-02-09T22:17:10.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:17:10.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:17:10.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:17:10.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:17:10.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-02-09T22:17:10.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:17:10.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:17:10.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef9769f2-c34e-4bf6-8909-e8925282d5b9 2024-02-09T22:17:10.228 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 ef9769f2-c34e-4bf6-8909-e8925282d5b9 2024-02-09T22:17:10.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ef9769f2-c34e-4bf6-8909-e8925282d5b9' 2024-02-09T22:17:10.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:17:10.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmpMZl3LY0DhAAslrlMUGyyDcQjAl4QSlmnQ== 2024-02-09T22:17:10.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmpMZl3LY0DhAAslrlMUGyyDcQjAl4QSlmnQ=="}' 2024-02-09T22:17:10.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef9769f2-c34e-4bf6-8909-e8925282d5b9 -i td/osd-recovery-scrub/1/new.json 2024-02-09T22:17:10.592 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:17:10.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-02-09T22:17:10.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBmpMZl3LY0DhAAslrlMUGyyDcQjAl4QSlmnQ== --osd-uuid ef9769f2-c34e-4bf6-8909-e8925282d5b9 2024-02-09T22:17:10.633 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.630+0000 7f76bd7fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:10.661 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.658+0000 7f76bd7fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.659+0000 7f76bd7fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.659+0000 7f76bd7fa640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-09T22:17:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.659+0000 7f76bd7fa640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-09T22:17:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.659+0000 7f76bd7fa640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-09T22:17:10.727 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:10.724+0000 7f76bd7fa640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-02-09T22:17:15.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-02-09T22:17:15.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:17:15.869 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-02-09T22:17:15.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-09T22:17:15.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:17:16.221 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-02-09T22:17:16.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-09T22:17:16.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-09T22:17:16.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:17:16.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:17:16.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:17:16.247 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:16.243+0000 7fb55b973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:16.249 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:16.245+0000 7fb55b973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:16.250 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:16.246+0000 7fb55b973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-09T22:17:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:17:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-09T22:17:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:17:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:17:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:16.493 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:17:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:17:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:17:16.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:16.783 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:16.780+0000 7fb55b973640 -1 Falling back to public interface 2024-02-09T22:17:17.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:17.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:17.762 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:17:17.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:17:17.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:17.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:17:18.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:19.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:19.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:19.072 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:17:19.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:17:19.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:19.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:17:19.249 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:19.245+0000 7fb55b973640 -1 osd.1 0 log_to_monitors true 2024-02-09T22:17:19.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:20.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:20.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:20.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:17:20.366 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:17:20.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:20.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:17:20.639 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/614801183,v1:127.0.0.1:6811/614801183] [v2:127.0.0.1:6812/614801183,v1:127.0.0.1:6813/614801183] exists,up ef9769f2-c34e-4bf6-8909-e8925282d5b9 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:17:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:17:20.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:17:20.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad6896d5-015d-4986-8327-29003117152f 2024-02-09T22:17:20.646 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 ad6896d5-015d-4986-8327-29003117152f 2024-02-09T22:17:20.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ad6896d5-015d-4986-8327-29003117152f' 2024-02-09T22:17:20.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:17:20.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwpMZl3sI4JxAAspWVzaoAnaxmP57SWhxuqQ== 2024-02-09T22:17:20.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwpMZl3sI4JxAAspWVzaoAnaxmP57SWhxuqQ=="}' 2024-02-09T22:17:20.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad6896d5-015d-4986-8327-29003117152f -i td/osd-recovery-scrub/2/new.json 2024-02-09T22:17:21.002 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:17:21.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-02-09T22:17:21.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBwpMZl3sI4JxAAspWVzaoAnaxmP57SWhxuqQ== --osd-uuid ad6896d5-015d-4986-8327-29003117152f 2024-02-09T22:17:21.043 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.039+0000 7f7cd51df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:21.087 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.083+0000 7f7cd51df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:21.088 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.084+0000 7f7cd51df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:21.088 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.085+0000 7f7cd51df640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-09T22:17:21.088 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.085+0000 7f7cd51df640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-09T22:17:21.088 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.085+0000 7f7cd51df640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-09T22:17:21.153 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:21.150+0000 7f7cd51df640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-02-09T22:17:26.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-02-09T22:17:26.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:17:26.626 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-02-09T22:17:26.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-09T22:17:26.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:17:26.982 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-02-09T22:17:26.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-09T22:17:26.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-09T22:17:26.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:17:26.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:17:26.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:17:27.009 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:27.005+0000 7fb2100b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:27.010 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:27.007+0000 7fb2100b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:27.011 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:27.008+0000 7fb2100b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:27.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-09T22:17:27.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:17:27.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-09T22:17:27.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:17:27.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:17:27.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:27.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:17:27.255 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:17:27.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:27.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:17:27.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:27.533 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:27.530+0000 7fb2100b5640 -1 Falling back to public interface 2024-02-09T22:17:28.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:28.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:28.525 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:17:28.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:17:28.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:28.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:17:28.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:29.717 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:29.713+0000 7fb2100b5640 -1 osd.2 0 log_to_monitors true 2024-02-09T22:17:29.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:29.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:29.805 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:17:29.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:17:29.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:29.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:17:30.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:31.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:31.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:31.103 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:17:31.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:17:31.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:31.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:17:31.377 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3673002966,v1:127.0.0.1:6819/3673002966] [v2:127.0.0.1:6820/3673002966,v1:127.0.0.1:6821/3673002966] exists,up ad6896d5-015d-4986-8327-29003117152f 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-02-09T22:17:31.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:17:31.383 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-09T22:17:31.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-02-09T22:17:31.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:17:31.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f99a095-b191-41df-a20f-5173df4c83b0 2024-02-09T22:17:31.385 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 4f99a095-b191-41df-a20f-5173df4c83b0 2024-02-09T22:17:31.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 4f99a095-b191-41df-a20f-5173df4c83b0' 2024-02-09T22:17:31.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:17:31.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7pMZlTTiWFxAARx4SzM2O2cwgdWWbq10ajA== 2024-02-09T22:17:31.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7pMZlTTiWFxAARx4SzM2O2cwgdWWbq10ajA=="}' 2024-02-09T22:17:31.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f99a095-b191-41df-a20f-5173df4c83b0 -i td/osd-recovery-scrub/3/new.json 2024-02-09T22:17:31.752 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:17:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-02-09T22:17:31.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB7pMZlTTiWFxAARx4SzM2O2cwgdWWbq10ajA== --osd-uuid 4f99a095-b191-41df-a20f-5173df4c83b0 2024-02-09T22:17:31.792 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.789+0000 7fc4b2813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:31.820 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.817+0000 7fc4b2813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:31.821 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.818+0000 7fc4b2813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:31.822 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.818+0000 7fc4b2813640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-09T22:17:31.822 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.818+0000 7fc4b2813640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-09T22:17:31.822 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.818+0000 7fc4b2813640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-09T22:17:31.895 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:31.892+0000 7fc4b2813640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-02-09T22:17:37.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-02-09T22:17:37.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:17:37.410 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-02-09T22:17:37.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-09T22:17:37.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:17:37.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-09T22:17:37.760 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-02-09T22:17:37.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-09T22:17:37.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:17:37.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:17:37.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:17:37.786 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:37.782+0000 7f1373811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:37.787 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:37.784+0000 7f1373811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:37.788 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:37.785+0000 7f1373811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:17:38.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-09T22:17:38.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:17:38.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-09T22:17:38.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:17:38.056 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:17:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:17:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:17:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:17:38.316 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:38.313+0000 7f1373811640 -1 Falling back to public interface 2024-02-09T22:17:38.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:17:39.332 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:17:39.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:39.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:17:39.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:40.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:40.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:40.614 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:17:40.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:17:40.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:40.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:17:40.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:17:40.657+0000 7f1373811640 -1 osd.3 0 log_to_monitors true 2024-02-09T22:17:40.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:17:41.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:17:41.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:17:41.893 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:17:41.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:17:41.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:17:41.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:17:42.200 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2929835574,v1:127.0.0.1:6827/2929835574] [v2:127.0.0.1:6828/2929835574,v1:127.0.0.1:6829/2929835574] exists,up 4f99a095-b191-41df-a20f-5173df4c83b0 2024-02-09T22:17:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:17:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:17:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:17:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-02-09T22:17:42.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-09T22:17:42.598 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' created 2024-02-09T22:17:42.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-09T22:17:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-02-09T22:17:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-09T22:17:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-09T22:17:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-09T22:17:43.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-09T22:17:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-09T22:17:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-09T22:17:43.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-09T22:17:43.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-09T22:17:43.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-09T22:17:43.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-09T22:17:43.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-09T22:17:43.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-09T22:17:43.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-09T22:17:43.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-09T22:17:43.703 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-09T22:17:44.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-09T22:17:44.060 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-02-09T22:17:44.060 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-02-09T22:17:44.060 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-02-09T22:17:44.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-09T22:17:44.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:17:44.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-09T22:17:44.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-02-09T22:17:44.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-02-09T22:17:44.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-02-09T22:17:44.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:17:44.173 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2024-02-09T22:17:44.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:17:44.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-09T22:17:44.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-02-09T22:17:44.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-02-09T22:17:44.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2024-02-09T22:17:44.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:17:44.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-09T22:17:44.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-02-09T22:17:44.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-02-09T22:17:44.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444 3-85899345922' 2024-02-09T22:17:44.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:17:44.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-02-09T22:17:44.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:17:44.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-09T22:17:44.494 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-02-09T22:17:44.494 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:17:44.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-02-09T22:17:44.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-02-09T22:17:44.496 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836488 2024-02-09T22:17:44.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-09T22:17:44.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836488 2024-02-09T22:17:44.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-09T22:17:45.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-09T22:17:45.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-09T22:17:46.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836488 2024-02-09T22:17:46.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:17:46.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-09T22:17:46.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:17:46.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-09T22:17:46.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-09T22:17:46.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:17:46.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-09T22:17:46.108 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672966 2024-02-09T22:17:46.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-09T22:17:46.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-09T22:17:46.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-02-09T22:17:46.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:17:46.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-02-09T22:17:46.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:17:46.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-09T22:17:46.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-02-09T22:17:46.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:17:46.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-02-09T22:17:46.423 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509444 2024-02-09T22:17:46.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-02-09T22:17:46.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-09T22:17:46.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-02-09T22:17:46.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:17:46.695 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-02-09T22:17:46.695 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:17:46.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-09T22:17:46.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-02-09T22:17:46.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:17:46.698 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345922 2024-02-09T22:17:46.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-02-09T22:17:46.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-02-09T22:17:46.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-09T22:17:46.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-02-09T22:17:46.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-09T22:17:46.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-09T22:17:46.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-09T22:17:47.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-09T22:17:47.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-09T22:17:47.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-09T22:17:47.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-09T22:17:47.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-09T22:17:47.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-09T22:17:47.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-09T22:17:47.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-09T22:17:47.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-09T22:17:47.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-09T22:17:47.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-09T22:17:47.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-09T22:17:47.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-09T22:17:47.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-09T22:17:47.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-09T22:17:47.912 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-02-09T22:17:47.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-02-09T22:17:47.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-02-09T22:17:48.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-02-09T22:17:48.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-02-09T22:17:48.491 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:17:48.491 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:17:44.018351+0000 0'0 23:8 [1] 1 [1] 1 0'0 2024-02-09T22:17:42.547280+0000 0'0 2024-02-09T22:17:42.547280+0000 0 0 periodic scrub scheduled @ 2024-02-11T01:34:52.328946+0000 0 0 2024-02-09T22:17:48.491 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:17:48.491 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:17:48.491 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:17:48.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.44749 bs=1M count=50 2024-02-09T22:17:48.642 INFO:tasks.workunit.client.0.smithi134.stderr:50+0 records in 2024-02-09T22:17:48.642 INFO:tasks.workunit.client.0.smithi134.stderr:50+0 records out 2024-02-09T22:17:48.643 INFO:tasks.workunit.client.0.smithi134.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.139733 s, 375 MB/s 2024-02-09T22:17:48.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-02-09T22:17:48.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:48.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.44749 2024-02-09T22:17:50.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:50.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.44749 2024-02-09T22:17:51.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:51.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.44749 2024-02-09T22:17:53.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:53.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.44749 2024-02-09T22:17:54.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:54.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.44749 2024-02-09T22:17:55.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:55.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.44749 2024-02-09T22:17:57.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:57.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.44749 2024-02-09T22:17:58.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:17:58.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.44749 2024-02-09T22:18:00.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:00.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.44749 2024-02-09T22:18:01.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:01.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.44749 2024-02-09T22:18:03.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:03.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.44749 2024-02-09T22:18:04.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:04.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.44749 2024-02-09T22:18:05.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:05.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.44749 2024-02-09T22:18:07.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:07.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.44749 2024-02-09T22:18:08.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:08.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.44749 2024-02-09T22:18:10.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:10.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.44749 2024-02-09T22:18:11.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:11.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.44749 2024-02-09T22:18:12.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:12.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.44749 2024-02-09T22:18:14.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:14.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.44749 2024-02-09T22:18:15.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:15.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.44749 2024-02-09T22:18:17.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:17.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.44749 2024-02-09T22:18:18.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:18.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.44749 2024-02-09T22:18:20.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:20.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.44749 2024-02-09T22:18:21.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:21.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.44749 2024-02-09T22:18:23.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:23.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.44749 2024-02-09T22:18:24.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:24.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.44749 2024-02-09T22:18:25.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:25.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.44749 2024-02-09T22:18:27.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:27.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.44749 2024-02-09T22:18:28.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:28.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.44749 2024-02-09T22:18:30.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:30.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.44749 2024-02-09T22:18:31.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:31.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.44749 2024-02-09T22:18:32.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:32.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.44749 2024-02-09T22:18:34.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:34.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.44749 2024-02-09T22:18:35.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:35.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.44749 2024-02-09T22:18:37.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:37.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.44749 2024-02-09T22:18:38.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:38.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.44749 2024-02-09T22:18:39.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:39.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.44749 2024-02-09T22:18:41.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:41.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.44749 2024-02-09T22:18:42.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:42.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.44749 2024-02-09T22:18:44.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:44.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.44749 2024-02-09T22:18:45.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:45.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.44749 2024-02-09T22:18:46.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:46.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.44749 2024-02-09T22:18:48.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:48.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.44749 2024-02-09T22:18:49.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:49.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.44749 2024-02-09T22:18:51.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:51.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.44749 2024-02-09T22:18:52.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:52.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.44749 2024-02-09T22:18:53.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:53.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.44749 2024-02-09T22:18:55.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:55.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.44749 2024-02-09T22:18:56.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:56.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.44749 2024-02-09T22:18:58.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:58.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.44749 2024-02-09T22:18:59.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:18:59.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.44749 2024-02-09T22:19:01.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:01.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.44749 2024-02-09T22:19:02.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:02.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.44749 2024-02-09T22:19:04.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:04.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.44749 2024-02-09T22:19:05.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:05.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.44749 2024-02-09T22:19:06.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:06.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.44749 2024-02-09T22:19:08.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:08.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.44749 2024-02-09T22:19:09.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:09.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.44749 2024-02-09T22:19:11.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:11.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.44749 2024-02-09T22:19:12.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:12.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.44749 2024-02-09T22:19:14.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:14.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.44749 2024-02-09T22:19:15.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:15.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.44749 2024-02-09T22:19:17.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:17.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.44749 2024-02-09T22:19:18.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:18.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.44749 2024-02-09T22:19:20.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:20.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.44749 2024-02-09T22:19:21.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:21.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.44749 2024-02-09T22:19:23.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:23.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.44749 2024-02-09T22:19:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:24.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.44749 2024-02-09T22:19:26.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:26.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.44749 2024-02-09T22:19:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.44749 2024-02-09T22:19:28.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:28.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.44749 2024-02-09T22:19:30.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:30.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.44749 2024-02-09T22:19:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.44749 2024-02-09T22:19:33.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:33.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.44749 2024-02-09T22:19:34.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.44749 2024-02-09T22:19:36.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:36.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.44749 2024-02-09T22:19:37.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:37.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.44749 2024-02-09T22:19:39.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:39.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.44749 2024-02-09T22:19:41.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:41.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.44749 2024-02-09T22:19:42.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:42.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.44749 2024-02-09T22:19:44.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:44.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.44749 2024-02-09T22:19:45.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:45.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.44749 2024-02-09T22:19:47.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:47.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.44749 2024-02-09T22:19:48.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:48.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.44749 2024-02-09T22:19:49.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:49.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.44749 2024-02-09T22:19:51.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:51.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.44749 2024-02-09T22:19:52.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:52.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.44749 2024-02-09T22:19:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.44749 2024-02-09T22:19:55.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:55.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.44749 2024-02-09T22:19:57.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:57.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.44749 2024-02-09T22:19:58.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:19:58.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.44749 2024-02-09T22:20:00.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:00.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.44749 2024-02-09T22:20:01.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:01.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.44749 2024-02-09T22:20:02.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:02.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.44749 2024-02-09T22:20:04.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:04.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.44749 2024-02-09T22:20:05.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:05.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.44749 2024-02-09T22:20:07.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:07.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.44749 2024-02-09T22:20:08.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:08.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.44749 2024-02-09T22:20:10.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:10.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.44749 2024-02-09T22:20:11.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-09T22:20:11.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.44749 2024-02-09T22:20:13.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.44749 2024-02-09T22:20:13.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-02-09T22:20:13.633 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 4 2024-02-09T22:20:13.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-02-09T22:20:13.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-02-09T22:20:13.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-09T22:20:13.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-09T22:20:13.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-09T22:20:13.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-09T22:20:13.906 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:20:13.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-09T22:20:15.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-02-09T22:20:15.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-02-09T22:20:15.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-02-09T22:20:15.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-09T22:20:15.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-09T22:20:15.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-09T22:20:15.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-09T22:20:16.203 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:20:16.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-09T22:20:18.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-02-09T22:20:18.216 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-02-09T22:20:18.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-02-09T22:20:18.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-09T22:20:18.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-09T22:20:18.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-09T22:20:18.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-09T22:20:18.483 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:20:18.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-09T22:20:20.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-02-09T22:20:20.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-02-09T22:20:20.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-02-09T22:20:20.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-09T22:20:20.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-09T22:20:20.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-09T22:20:20.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-09T22:20:20.971 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:20:20.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-09T22:20:22.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-02-09T22:20:22.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-02-09T22:20:22.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-02-09T22:20:22.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-09T22:20:22.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-09T22:20:22.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-09T22:20:22.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-09T22:20:23.255 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:20:23.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-02-09T22:20:23.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-02-09T22:20:23.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-02-09T22:20:23.524 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:20:23.524 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-02-09T22:20:18.694757+0000 24'1300 27:1323 [1,0,2,3] 1 [1,0] 1 0'0 2024-02-09T22:17:42.547280+0000 0'0 2024-02-09T22:17:42.547280+0000 0 0 no scrub is scheduled 0 0 2024-02-09T22:20:23.524 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:20:23.524 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:20:23.524 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:20:23.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-02-09T22:20:33.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-02-09T22:20:33.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:20:33.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:20:33.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:20:33.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:20:33.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:20:38.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:20:38.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-02-09T22:20:38.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-02-09T22:20:38.853 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-02-09T22:20:38.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-02-09T22:20:38.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-02-09T22:20:38.858 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:26.470+0000 7ff344db6640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.858 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:33.482+0000 7ff344db6640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.858 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:37.516+0000 7ff344db6640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-02-09T22:20:38.870 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:23.847+0000 7fb554916640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.870 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:27.897+0000 7fb554916640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.871 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:28.898+0000 7fb554916640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.871 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:31.929+0000 7fb554916640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.871 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:34.936+0000 7fb554916640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.871 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:35.899+0000 7fb554916640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-02-09T22:20:38.873 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:19.915+0000 7fb209058640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.873 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:23.876+0000 7fb209058640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.873 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:24.917+0000 7fb209058640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.874 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:30.919+0000 7fb209058640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.874 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:31.962+0000 7fb209058640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.874 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:36.897+0000 7fb209058640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-02-09T22:20:38.876 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:19.467+0000 7f136c11c640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.876 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:21.455+0000 7f136c11c640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.876 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:23.433+0000 7f136c11c640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.876 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:30.448+0000 7f136c11c640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.876 INFO:tasks.workunit.client.0.smithi134.stdout:2024-02-09T22:20:36.506+0000 7f136c11c640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-09T22:20:38.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-02-09T22:20:38.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-02-09T22:20:38.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-02-09T22:20:38.877 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-02-09T22:20:38.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-02-09T22:20:38.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-02-09T22:20:38.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-09T22:20:38.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-02-09T22:20:38.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-02-09T22:20:38.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-02-09T22:20:38.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-09T22:20:38.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-02-09T22:20:38.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-02-09T22:20:38.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-02-09T22:20:38.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-09T22:20:38.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-02-09T22:20:38.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-02-09T22:20:38.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:38.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-02-09T22:20:38.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-09T22:20:38.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:20:38.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:20:38.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:20:38.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:20:38.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:20:38.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:20:38.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:20:38.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:20:38.912 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:20:38.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:20:38.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:20:38.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:20:38.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:20:38.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:20:38.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-09T22:20:38.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-09T22:20:38.937 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:20:38.937 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:38.937 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:38.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44749 2024-02-09T22:20:38.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:20:38.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:20:38.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-02-09T22:20:38.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-02-09T22:20:38.938 INFO:tasks.workunit.client.0.smithi134.stdout:TEST PASSED 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.44749 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-02-09T22:20:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:20:38.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:20:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:20:38.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:20:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:20:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:20:38.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:20:38.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:20:38.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:20:38.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:20:38.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:20:38.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:20:38.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-09T22:20:38.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-09T22:20:38.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:20:38.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:38.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:38.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44749 2024-02-09T22:20:38.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:20:38.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:20:38.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-02-09T22:20:38.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-09T22:20:38.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:38.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44749 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-02-09T22:20:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-09T22:20:38.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-09T22:20:38.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-09T22:20:38.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-02-09T22:20:38.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:20:39.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-09T22:20:39.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-09T22:20:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-09T22:20:39.529 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:39.529 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:39.529 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44749/ceph-mon.a.asok 2024-02-09T22:20:39.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-09T22:20:39.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44749/ceph-mon.a.asok config get fsid 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-09T22:20:39.607 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44749/ceph-mon.a.asok 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-09T22:20:39.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44749/ceph-mon.a.asok config get mon_host 2024-02-09T22:20:39.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-02-09T22:20:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-02-09T22:20:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-09T22:20:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-09T22:20:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-09T22:20:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-02-09T22:20:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-09T22:20:40.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-09T22:20:40.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:20:40.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:20:40.491 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:20:40.491 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:40.491 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:40.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:20:40.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-09T22:20:40.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-09T22:20:40.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-02-09T22:20:40.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-02-09T22:20:40.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-02-09T22:20:40.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-02-09T22:20:40.519 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-02-09T22:20:40.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:20:40.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:20:40.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-02-09T22:20:40.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:20:40.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56a77822-db8e-4e73-8312-bfa51fcc3f37 2024-02-09T22:20:40.527 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 56a77822-db8e-4e73-8312-bfa51fcc3f37 2024-02-09T22:20:40.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 56a77822-db8e-4e73-8312-bfa51fcc3f37' 2024-02-09T22:20:40.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:20:40.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4pcZljAgaIBAA5oAEOUgKFe2yyLxds0KZBg== 2024-02-09T22:20:40.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4pcZljAgaIBAA5oAEOUgKFe2yyLxds0KZBg=="}' 2024-02-09T22:20:40.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56a77822-db8e-4e73-8312-bfa51fcc3f37 -i td/osd-recovery-scrub/0/new.json 2024-02-09T22:20:40.755 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:20:40.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-02-09T22:20:40.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQA4pcZljAgaIBAA5oAEOUgKFe2yyLxds0KZBg== --osd-uuid 56a77822-db8e-4e73-8312-bfa51fcc3f37 2024-02-09T22:20:40.798 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.794+0000 7fd83440d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:40.806 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.803+0000 7fd83440d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:40.807 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.804+0000 7fd83440d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:40.808 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.804+0000 7fd83440d640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-09T22:20:40.808 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.804+0000 7fd83440d640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-09T22:20:40.808 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.804+0000 7fd83440d640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-09T22:20:40.889 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:40.886+0000 7fd83440d640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-02-09T22:20:46.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-02-09T22:20:46.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:20:46.469 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-02-09T22:20:46.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-09T22:20:46.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:20:46.825 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-02-09T22:20:46.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-09T22:20:46.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:20:46.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:20:46.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:20:46.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:20:46.855 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:46.852+0000 7f21bfaf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:46.857 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:46.854+0000 7f21bfaf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:46.858 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:46.855+0000 7f21bfaf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:47.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-09T22:20:47.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:20:47.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:20:47.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:20:47.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:20:47.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:47.372+0000 7f21bfaf5640 -1 Falling back to public interface 2024-02-09T22:20:48.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:20:48.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:20:48.369 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:20:48.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:20:48.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:20:48.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:20:48.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:20:49.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:20:49.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:20:49.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:20:49.642 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:20:49.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:20:49.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:20:49.752 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:49.748+0000 7f21bfaf5640 -1 osd.0 0 log_to_monitors true 2024-02-09T22:20:49.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:20:50.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:20:50.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:20:50.961 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:20:50.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:20:50.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:20:50.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-09T22:20:51.284 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3537596102,v1:127.0.0.1:6803/3537596102] [v2:127.0.0.1:6804/3537596102,v1:127.0.0.1:6805/3537596102] exists,up 56a77822-db8e-4e73-8312-bfa51fcc3f37 2024-02-09T22:20:51.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:20:51.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:20:51.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-02-09T22:20:51.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:20:51.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-02-09T22:20:51.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:20:51.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6f1790c-07f9-4d39-95d4-81397ada4934 2024-02-09T22:20:51.292 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 d6f1790c-07f9-4d39-95d4-81397ada4934 2024-02-09T22:20:51.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d6f1790c-07f9-4d39-95d4-81397ada4934' 2024-02-09T22:20:51.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:20:51.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDpcZlpvj7ERAAr93jaEDioWyJST4cBGgLLg== 2024-02-09T22:20:51.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDpcZlpvj7ERAAr93jaEDioWyJST4cBGgLLg=="}' 2024-02-09T22:20:51.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6f1790c-07f9-4d39-95d4-81397ada4934 -i td/osd-recovery-scrub/1/new.json 2024-02-09T22:20:51.696 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:20:51.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-02-09T22:20:51.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBDpcZlpvj7ERAAr93jaEDioWyJST4cBGgLLg== --osd-uuid d6f1790c-07f9-4d39-95d4-81397ada4934 2024-02-09T22:20:51.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.737+0000 7f378da11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:51.772 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.770+0000 7f378da11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:51.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.771+0000 7f378da11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:51.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.771+0000 7f378da11640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-09T22:20:51.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.771+0000 7f378da11640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-09T22:20:51.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.771+0000 7f378da11640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-09T22:20:51.839 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:51.837+0000 7f378da11640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-02-09T22:20:57.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-02-09T22:20:57.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:20:57.418 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-02-09T22:20:57.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-09T22:20:57.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:20:57.769 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-02-09T22:20:57.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-09T22:20:57.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:20:57.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:20:57.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:20:57.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:20:57.799 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:57.796+0000 7f0998613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:57.801 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:57.798+0000 7f0998613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:57.802 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:57.799+0000 7f0998613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:20:58.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-09T22:20:58.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:20:58.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-09T22:20:58.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:20:58.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:20:58.039 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:20:58.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:20:58.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:20:58.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:20:58.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:20:58.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:20:58.320 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:20:58.317+0000 7f0998613640 -1 Falling back to public interface 2024-02-09T22:20:59.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:20:59.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:20:59.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:20:59.309 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:20:59.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:20:59.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:20:59.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:00.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:00.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:00.606 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:00.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:21:00.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:00.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:21:00.750 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:00.747+0000 7f0998613640 -1 osd.1 0 log_to_monitors true 2024-02-09T22:21:00.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:01.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:01.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:01.881 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:01.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:21:01.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:01.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-09T22:21:02.162 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3777393808,v1:127.0.0.1:6811/3777393808] [v2:127.0.0.1:6812/3777393808,v1:127.0.0.1:6813/3777393808] exists,up d6f1790c-07f9-4d39-95d4-81397ada4934 2024-02-09T22:21:02.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:21:02.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:21:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:21:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1e0f57a8-7dcd-4196-921c-aa64598eb3ef 2024-02-09T22:21:02.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1e0f57a8-7dcd-4196-921c-aa64598eb3ef' 2024-02-09T22:21:02.170 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 1e0f57a8-7dcd-4196-921c-aa64598eb3ef 2024-02-09T22:21:02.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:21:02.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOpcZlF7u6ChAAW3yRFBA8uvwKAzfNep8tmw== 2024-02-09T22:21:02.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOpcZlF7u6ChAAW3yRFBA8uvwKAzfNep8tmw=="}' 2024-02-09T22:21:02.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1e0f57a8-7dcd-4196-921c-aa64598eb3ef -i td/osd-recovery-scrub/2/new.json 2024-02-09T22:21:02.538 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:02.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-02-09T22:21:02.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBOpcZlF7u6ChAAW3yRFBA8uvwKAzfNep8tmw== --osd-uuid 1e0f57a8-7dcd-4196-921c-aa64598eb3ef 2024-02-09T22:21:02.582 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.578+0000 7f3557415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:02.622 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.619+0000 7f3557415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:02.623 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.620+0000 7f3557415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:02.624 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.620+0000 7f3557415640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-09T22:21:02.624 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.620+0000 7f3557415640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-09T22:21:02.624 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.620+0000 7f3557415640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-09T22:21:02.698 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:02.695+0000 7f3557415640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-02-09T22:21:08.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-02-09T22:21:08.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:21:08.386 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-02-09T22:21:08.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-09T22:21:08.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:21:08.738 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-02-09T22:21:08.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-09T22:21:08.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:08.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:21:08.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:21:08.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:21:08.768 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:08.765+0000 7f8200ed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:08.770 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:08.767+0000 7f8200ed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:08.771 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:08.768+0000 7f8200ed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:09.012 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:21:09.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:21:09.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:09.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:21:09.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:09.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:09.293+0000 7f8200ed3640 -1 Falling back to public interface 2024-02-09T22:21:10.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:10.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:10.284 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:21:10.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:21:10.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:10.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:21:10.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:11.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:11.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:11.565 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:11.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:21:11.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:11.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:21:11.618 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:11.615+0000 7f8200ed3640 -1 osd.2 0 log_to_monitors true 2024-02-09T22:21:11.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:12.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:12.844 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:12.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:12.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:21:12.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:12.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-09T22:21:13.128 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3009744492,v1:127.0.0.1:6819/3009744492] [v2:127.0.0.1:6820/3009744492,v1:127.0.0.1:6821/3009744492] exists,up 1e0f57a8-7dcd-4196-921c-aa64598eb3ef 2024-02-09T22:21:13.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:21:13.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:21:13.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-02-09T22:21:13.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:21:13.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-02-09T22:21:13.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:21:13.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a0a6ebe-e304-468b-9b55-24c262616ff2 2024-02-09T22:21:13.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 4a0a6ebe-e304-468b-9b55-24c262616ff2' 2024-02-09T22:21:13.139 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 4a0a6ebe-e304-468b-9b55-24c262616ff2 2024-02-09T22:21:13.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:21:13.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZpcZl/YivCBAAIfItIH0nKyivlwdXjLEdMQ== 2024-02-09T22:21:13.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZpcZl/YivCBAAIfItIH0nKyivlwdXjLEdMQ=="}' 2024-02-09T22:21:13.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a0a6ebe-e304-468b-9b55-24c262616ff2 -i td/osd-recovery-scrub/3/new.json 2024-02-09T22:21:13.512 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:13.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-02-09T22:21:13.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBZpcZl/YivCBAAIfItIH0nKyivlwdXjLEdMQ== --osd-uuid 4a0a6ebe-e304-468b-9b55-24c262616ff2 2024-02-09T22:21:13.556 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.553+0000 7fa436615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:13.596 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.593+0000 7fa436615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:13.597 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.594+0000 7fa436615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:13.598 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.594+0000 7fa436615640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-09T22:21:13.598 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.594+0000 7fa436615640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-09T22:21:13.598 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.594+0000 7fa436615640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-09T22:21:13.671 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:13.668+0000 7fa436615640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-02-09T22:21:19.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-02-09T22:21:19.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:21:19.654 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-02-09T22:21:19.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-09T22:21:19.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:21:20.004 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-02-09T22:21:20.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-09T22:21:20.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:21:20.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:20.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:21:20.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:21:20.034 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:20.030+0000 7fe9a1013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:20.036 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:20.032+0000 7fe9a1013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:20.037 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:20.033+0000 7fe9a1013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:20.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:21:20.275 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:21:20.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:20.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:21:20.562 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:20.559+0000 7fe9a1013640 -1 Falling back to public interface 2024-02-09T22:21:20.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:21.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:21.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:21.565 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:21:21.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:21:21.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:21.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:21:21.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:22.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:21:22.842 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:22.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:22.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:21:22.978 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:22.974+0000 7fe9a1013640 -1 osd.3 0 log_to_monitors true 2024-02-09T22:21:23.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:24.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:24.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:24.179 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:24.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:21:24.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:24.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-09T22:21:24.460 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2940388851,v1:127.0.0.1:6827/2940388851] [v2:127.0.0.1:6828/2940388851,v1:127.0.0.1:6829/2940388851] exists,up 4a0a6ebe-e304-468b-9b55-24c262616ff2 2024-02-09T22:21:24.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:21:24.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:21:24.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:21:24.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:21:24.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-02-09T22:21:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:21:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5abcf227-f3b9-49bf-9339-73eb25ca06f5 2024-02-09T22:21:24.467 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 5abcf227-f3b9-49bf-9339-73eb25ca06f5 2024-02-09T22:21:24.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 5abcf227-f3b9-49bf-9339-73eb25ca06f5' 2024-02-09T22:21:24.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:21:24.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkpcZlBOqMHBAAy5bDn2FmtkD8oV5Liy6k3A== 2024-02-09T22:21:24.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkpcZlBOqMHBAAy5bDn2FmtkD8oV5Liy6k3A=="}' 2024-02-09T22:21:24.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5abcf227-f3b9-49bf-9339-73eb25ca06f5 -i td/osd-recovery-scrub/4/new.json 2024-02-09T22:21:24.915 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-02-09T22:21:24.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-02-09T22:21:24.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBkpcZlBOqMHBAAy5bDn2FmtkD8oV5Liy6k3A== --osd-uuid 5abcf227-f3b9-49bf-9339-73eb25ca06f5 2024-02-09T22:21:24.959 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:24.955+0000 7fa42160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:24.991 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:24.988+0000 7fa42160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:24.992 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:24.989+0000 7fa42160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:24.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:24.989+0000 7fa42160f640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-09T22:21:24.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:24.990+0000 7fa42160f640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-09T22:21:24.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:24.990+0000 7fa42160f640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-09T22:21:25.078 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:25.075+0000 7fa42160f640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-02-09T22:21:31.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-02-09T22:21:31.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:21:31.076 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-02-09T22:21:31.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-09T22:21:31.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:21:31.431 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-02-09T22:21:31.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-09T22:21:31.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:31.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:21:31.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:21:31.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:21:31.461 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:31.458+0000 7ff599218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:31.463 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:31.460+0000 7ff599218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:31.464 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:31.461+0000 7ff599218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:31.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-09T22:21:31.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:21:31.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-09T22:21:31.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:21:31.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:21:31.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:31.705 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:21:31.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:21:31.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:31.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-09T22:21:31.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:31.990 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:31.987+0000 7ff599218640 -1 Falling back to public interface 2024-02-09T22:21:32.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:32.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:32.975 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:21:32.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:21:32.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:32.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-09T22:21:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:34.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:34.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:34.250 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:21:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-09T22:21:34.328 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:34.326+0000 7ff599218640 -1 osd.4 0 log_to_monitors true 2024-02-09T22:21:34.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:35.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:35.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:35.534 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:35.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:21:35.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:35.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-09T22:21:35.815 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1999477482,v1:127.0.0.1:6835/1999477482] [v2:127.0.0.1:6836/1999477482,v1:127.0.0.1:6837/1999477482] exists,up 5abcf227-f3b9-49bf-9339-73eb25ca06f5 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-02-09T22:21:35.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:21:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:21:35.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c621334-9620-4166-93de-070a28772eaf 2024-02-09T22:21:35.823 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 0c621334-9620-4166-93de-070a28772eaf 2024-02-09T22:21:35.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 0c621334-9620-4166-93de-070a28772eaf' 2024-02-09T22:21:35.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:21:35.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvpcZlxRegMRAA70nV2Kiz9avUVW35UzE0wA== 2024-02-09T22:21:35.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvpcZlxRegMRAA70nV2Kiz9avUVW35UzE0wA=="}' 2024-02-09T22:21:35.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c621334-9620-4166-93de-070a28772eaf -i td/osd-recovery-scrub/5/new.json 2024-02-09T22:21:36.190 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-02-09T22:21:36.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-02-09T22:21:36.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBvpcZlxRegMRAA70nV2Kiz9avUVW35UzE0wA== --osd-uuid 0c621334-9620-4166-93de-070a28772eaf 2024-02-09T22:21:36.234 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.230+0000 7f12f499e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:36.240 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.237+0000 7f12f499e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:36.241 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.238+0000 7f12f499e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:36.242 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.239+0000 7f12f499e640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-09T22:21:36.242 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.239+0000 7f12f499e640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-09T22:21:36.242 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.239+0000 7f12f499e640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-09T22:21:36.349 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:36.346+0000 7f12f499e640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-02-09T22:21:42.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-02-09T22:21:42.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:21:42.299 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-02-09T22:21:42.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-09T22:21:42.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:21:42.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-09T22:21:42.741 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-02-09T22:21:42.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:42.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:21:42.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:21:42.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:21:42.771 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:42.768+0000 7fa2aa6b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:42.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:42.770+0000 7fa2aa6b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:42.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:42.771+0000 7fa2aa6b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:43.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-09T22:21:43.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:21:43.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-09T22:21:43.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:21:43.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:21:43.053 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:21:43.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:43.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:21:43.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:43.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-09T22:21:43.289 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:43.286+0000 7fa2aa6b5640 -1 Falling back to public interface 2024-02-09T22:21:43.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:44.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:44.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:44.327 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:21:44.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:21:44.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:44.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-09T22:21:44.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:45.626 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:45.622+0000 7fa2aa6b5640 -1 osd.5 0 log_to_monitors true 2024-02-09T22:21:45.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:45.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:45.638 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:45.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:21:45.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:45.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-09T22:21:45.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:46.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:46.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:46.944 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:46.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:21:46.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:46.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-09T22:21:47.251 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2627167527,v1:127.0.0.1:6843/2627167527] [v2:127.0.0.1:6844/2627167527,v1:127.0.0.1:6845/2627167527] exists,up 0c621334-9620-4166-93de-070a28772eaf 2024-02-09T22:21:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:21:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:21:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:47.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5df66d44-09f2-4925-aa39-d5581dad8c6d 2024-02-09T22:21:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 5df66d44-09f2-4925-aa39-d5581dad8c6d' 2024-02-09T22:21:47.259 INFO:tasks.workunit.client.0.smithi134.stdout:add osd6 5df66d44-09f2-4925-aa39-d5581dad8c6d 2024-02-09T22:21:47.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:21:47.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7pcZlLfYHEBAA9tzVvAKOAwCsAvCKGpJc/w== 2024-02-09T22:21:47.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7pcZlLfYHEBAA9tzVvAKOAwCsAvCKGpJc/w=="}' 2024-02-09T22:21:47.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5df66d44-09f2-4925-aa39-d5581dad8c6d -i td/osd-recovery-scrub/6/new.json 2024-02-09T22:21:47.617 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-02-09T22:21:47.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-02-09T22:21:47.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB7pcZlLfYHEBAA9tzVvAKOAwCsAvCKGpJc/w== --osd-uuid 5df66d44-09f2-4925-aa39-d5581dad8c6d 2024-02-09T22:21:47.661 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.658+0000 7fa39daa6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:47.685 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.681+0000 7fa39daa6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:47.686 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.682+0000 7fa39daa6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:47.686 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.683+0000 7fa39daa6640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-09T22:21:47.686 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.683+0000 7fa39daa6640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-09T22:21:47.686 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.683+0000 7fa39daa6640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-09T22:21:47.760 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:47.756+0000 7fa39daa6640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-02-09T22:21:53.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-02-09T22:21:53.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:21:53.794 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd6 key to auth repository 2024-02-09T22:21:53.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-09T22:21:53.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:21:54.145 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.6 2024-02-09T22:21:54.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-09T22:21:54.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:54.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:21:54.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:21:54.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:21:54.175 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:54.171+0000 7f20aa262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:54.177 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:54.173+0000 7f20aa262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:54.178 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:54.174+0000 7f20aa262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:54.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-09T22:21:54.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:21:54.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-09T22:21:54.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:21:54.419 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:21:54.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:21:54.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:54.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:21:54.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:54.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-09T22:21:54.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:54.706+0000 7f20aa262640 -1 Falling back to public interface 2024-02-09T22:21:54.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:55.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:55.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:55.726 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:21:55.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:21:55.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:55.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-09T22:21:56.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:57.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:57.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:57.010 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:21:57.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:21:57.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:57.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-09T22:21:57.129 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:57.126+0000 7f20aa262640 -1 osd.6 0 log_to_monitors true 2024-02-09T22:21:57.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:21:58.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:21:58.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:21:58.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:21:58.332 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:21:58.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:21:58.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-09T22:21:58.644 INFO:tasks.workunit.client.0.smithi134.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2215899011,v1:127.0.0.1:6851/2215899011] [v2:127.0.0.1:6852/2215899011,v1:127.0.0.1:6853/2215899011] exists,up 5df66d44-09f2-4925-aa39-d5581dad8c6d 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-09T22:21:58.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-09T22:21:58.650 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-09T22:21:58.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ffd8d03a-922c-42d0-8929-4c916c9838ac 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 ffd8d03a-922c-42d0-8929-4c916c9838ac' 2024-02-09T22:21:58.652 INFO:tasks.workunit.client.0.smithi134.stdout:add osd7 ffd8d03a-922c-42d0-8929-4c916c9838ac 2024-02-09T22:21:58.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-09T22:21:58.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGpcZl8hR9JxAAmWLciYGatDIFdXNh5Z4Lug== 2024-02-09T22:21:58.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGpcZl8hR9JxAAmWLciYGatDIFdXNh5Z4Lug=="}' 2024-02-09T22:21:58.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ffd8d03a-922c-42d0-8929-4c916c9838ac -i td/osd-recovery-scrub/7/new.json 2024-02-09T22:21:59.028 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-02-09T22:21:59.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-02-09T22:21:59.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCGpcZl8hR9JxAAmWLciYGatDIFdXNh5Z4Lug== --osd-uuid ffd8d03a-922c-42d0-8929-4c916c9838ac 2024-02-09T22:21:59.071 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.068+0000 7fdc2836f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.142+0000 7fdc2836f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.144+0000 7fdc2836f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:21:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.144+0000 7fdc2836f640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-09T22:21:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.144+0000 7fdc2836f640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-09T22:21:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.144+0000 7fdc2836f640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-09T22:21:59.212 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:21:59.209+0000 7fdc2836f640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-02-09T22:22:05.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-02-09T22:22:05.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-09T22:22:05.567 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd7 key to auth repository 2024-02-09T22:22:05.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-09T22:22:05.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-09T22:22:05.920 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.7 2024-02-09T22:22:05.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-09T22:22:05.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=15e46a01-0635-4cb6-86cd-1d3c4762a886 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44749/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-09T22:22:05.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-09T22:22:05.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-09T22:22:05.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-09T22:22:05.950 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:22:05.947+0000 7f5559e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:22:05.952 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:22:05.949+0000 7f5559e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:22:05.953 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:22:05.950+0000 7f5559e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-09T22:22:06.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-09T22:22:06.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-09T22:22:06.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-09T22:22:06.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-09T22:22:06.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-09T22:22:06.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:22:06.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-09T22:22:06.192 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-02-09T22:22:06.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:22:06.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-09T22:22:06.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:22:06.485 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:22:06.482+0000 7f5559e1c640 -1 Falling back to public interface 2024-02-09T22:22:07.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:22:07.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:22:07.463 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-02-09T22:22:07.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-09T22:22:07.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:22:07.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-09T22:22:07.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:22:08.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:22:08.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:22:08.746 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-02-09T22:22:08.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-09T22:22:08.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:22:08.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-09T22:22:08.968 INFO:tasks.workunit.client.0.smithi134.stderr:2024-02-09T22:22:08.965+0000 7f5559e1c640 -1 osd.7 0 log_to_monitors true 2024-02-09T22:22:09.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-09T22:22:10.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-09T22:22:10.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-09T22:22:10.070 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-02-09T22:22:10.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-09T22:22:10.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-09T22:22:10.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-09T22:22:10.386 INFO:tasks.workunit.client.0.smithi134.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2496214404,v1:127.0.0.1:6859/2496214404] [v2:127.0.0.1:6860/2496214404,v1:127.0.0.1:6861/2496214404] exists,up ffd8d03a-922c-42d0-8929-4c916c9838ac 2024-02-09T22:22:10.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-09T22:22:10.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-09T22:22:10.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-09T22:22:10.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-02-09T22:22:10.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-02-09T22:22:10.784 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' created 2024-02-09T22:22:10.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-09T22:22:11.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-02-09T22:22:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-09T22:22:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-09T22:22:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-09T22:22:11.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-09T22:22:11.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-09T22:22:11.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-09T22:22:11.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-09T22:22:11.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-09T22:22:11.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-09T22:22:11.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-09T22:22:11.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-09T22:22:11.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-09T22:22:11.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-09T22:22:11.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-09T22:22:11.882 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-09T22:22:12.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-09T22:22:12.374 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-02-09T22:22:12.374 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:5 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:6 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:7' 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:12.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-09T22:22:12.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-02-09T22:22:12.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-02-09T22:22:12.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-02-09T22:22:12.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:12.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-09T22:22:12.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-09T22:22:12.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-09T22:22:12.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-02-09T22:22:12.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:12.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-09T22:22:13.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-09T22:22:13.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-09T22:22:13.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453' 2024-02-09T22:22:13.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:13.107 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-09T22:22:13.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-02-09T22:22:13.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-02-09T22:22:13.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931' 2024-02-09T22:22:13.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:13.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-09T22:22:13.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-02-09T22:22:13.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-02-09T22:22:13.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409' 2024-02-09T22:22:13.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:13.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-09T22:22:13.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-02-09T22:22:13.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-02-09T22:22:13.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2024-02-09T22:22:13.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:13.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-09T22:22:13.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-02-09T22:22:13.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-02-09T22:22:13.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2024-02-09T22:22:13.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-09T22:22:13.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-09T22:22:13.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-09T22:22:13.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-09T22:22:13.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2024-02-09T22:22:13.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:13.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-02-09T22:22:13.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:13.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-09T22:22:13.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-02-09T22:22:13.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:13.940 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836498 2024-02-09T22:22:13.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-02-09T22:22:13.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-02-09T22:22:13.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-09T22:22:14.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-02-09T22:22:14.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-09T22:22:15.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-09T22:22:15.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-09T22:22:15.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-02-09T22:22:15.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:15.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-09T22:22:15.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:15.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-09T22:22:15.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-09T22:22:15.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:15.497 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672976 2024-02-09T22:22:15.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-09T22:22:15.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-09T22:22:15.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-09T22:22:15.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-09T22:22:15.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:15.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-09T22:22:15.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:15.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-09T22:22:15.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-09T22:22:15.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:15.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-09T22:22:15.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-09T22:22:15.813 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509453 2024-02-09T22:22:15.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-09T22:22:16.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-02-09T22:22:16.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:16.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-02-09T22:22:16.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:16.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-09T22:22:16.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-02-09T22:22:16.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:16.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-02-09T22:22:16.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-02-09T22:22:16.085 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345931 2024-02-09T22:22:16.085 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-09T22:22:16.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345931 2024-02-09T22:22:16.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:16.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-02-09T22:22:16.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-09T22:22:16.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-02-09T22:22:16.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:16.359 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182409 2024-02-09T22:22:16.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-02-09T22:22:16.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-02-09T22:22:16.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-09T22:22:16.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182410 -lt 107374182409 2024-02-09T22:22:16.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:16.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-02-09T22:22:16.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:16.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-09T22:22:16.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-02-09T22:22:16.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:16.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-02-09T22:22:16.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-02-09T22:22:16.645 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018887 2024-02-09T22:22:16.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-09T22:22:16.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-02-09T22:22:16.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:16.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2024-02-09T22:22:16.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:16.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-09T22:22:16.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2024-02-09T22:22:16.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:16.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-02-09T22:22:16.920 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.6 seq 150323855365 2024-02-09T22:22:16.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-02-09T22:22:16.920 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-09T22:22:17.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-02-09T22:22:17.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-09T22:22:17.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-02-09T22:22:17.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-09T22:22:17.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-09T22:22:17.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-02-09T22:22:17.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-09T22:22:17.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-09T22:22:17.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-02-09T22:22:17.198 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.7 seq 171798691842 2024-02-09T22:22:17.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-09T22:22:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-02-09T22:22:17.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-09T22:22:17.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-09T22:22:17.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-09T22:22:17.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-09T22:22:18.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=30 2024-02-09T22:22:18.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-09T22:22:18.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-09T22:22:18.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-09T22:22:18.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 30 = 32 2024-02-09T22:22:18.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 30 '!=' -1 2024-02-09T22:22:18.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-09T22:22:18.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=30 2024-02-09T22:22:18.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-09T22:22:18.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-09T22:22:18.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-09T22:22:18.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-02-09T22:22:18.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-09T22:22:18.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-09T22:22:18.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-09T22:22:19.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-02-09T22:22:19.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-09T22:22:19.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-09T22:22:19.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-02-09T22:22:19.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-02-09T22:22:19.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-02-09T22:22:19.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-02-09T22:22:19.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.44749 bs=1M count=50 2024-02-09T22:22:19.542 INFO:tasks.workunit.client.0.smithi134.stderr:50+0 records in 2024-02-09T22:22:19.542 INFO:tasks.workunit.client.0.smithi134.stderr:50+0 records out 2024-02-09T22:22:19.543 INFO:tasks.workunit.client.0.smithi134.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.142027 s, 369 MB/s 2024-02-09T22:22:19.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-02-09T22:22:19.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:19.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.44749 2024-02-09T22:22:21.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:21.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.44749 2024-02-09T22:22:22.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:22.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.44749 2024-02-09T22:22:24.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:24.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.44749 2024-02-09T22:22:25.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:25.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.44749 2024-02-09T22:22:27.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:27.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.44749 2024-02-09T22:22:28.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:28.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.44749 2024-02-09T22:22:30.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:30.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.44749 2024-02-09T22:22:31.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:31.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.44749 2024-02-09T22:22:33.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:33.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.44749 2024-02-09T22:22:34.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:34.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.44749 2024-02-09T22:22:35.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:35.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.44749 2024-02-09T22:22:37.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:37.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.44749 2024-02-09T22:22:38.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:38.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.44749 2024-02-09T22:22:40.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:40.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.44749 2024-02-09T22:22:41.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:41.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.44749 2024-02-09T22:22:43.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:43.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.44749 2024-02-09T22:22:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.44749 2024-02-09T22:22:46.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:46.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.44749 2024-02-09T22:22:48.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:48.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.44749 2024-02-09T22:22:49.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:49.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.44749 2024-02-09T22:22:51.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:51.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.44749 2024-02-09T22:22:52.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:52.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.44749 2024-02-09T22:22:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.44749 2024-02-09T22:22:55.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:55.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.44749 2024-02-09T22:22:57.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:57.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.44749 2024-02-09T22:22:58.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:22:58.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.44749 2024-02-09T22:23:00.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:00.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.44749 2024-02-09T22:23:01.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:01.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.44749 2024-02-09T22:23:03.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:03.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.44749 2024-02-09T22:23:04.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:04.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.44749 2024-02-09T22:23:06.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:06.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.44749 2024-02-09T22:23:07.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:07.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.44749 2024-02-09T22:23:09.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:09.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.44749 2024-02-09T22:23:10.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:10.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.44749 2024-02-09T22:23:11.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:11.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.44749 2024-02-09T22:23:13.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:13.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.44749 2024-02-09T22:23:14.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:14.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.44749 2024-02-09T22:23:16.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:16.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.44749 2024-02-09T22:23:18.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-09T22:23:18.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.44749 2024-02-09T22:23:19.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.44749 2024-02-09T22:23:19.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-02-09T22:23:19.888 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-02-09T22:23:19.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-02-09T22:23:20.161 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-09T22:23:20.161 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-09T22:22:12.394791+0000 47'26 47:55 [5] 5 [5] 5 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.161 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.899991+0000 47'13 47:42 [3] 3 [3] 3 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.881666+0000 47'13 47:28 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.780937+0000 47'13 47:31 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.780959+0000 47'13 47:31 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:22:12.781175+0000 0'0 47:18 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-09T22:22:12.394530+0000 47'26 47:46 [5] 5 [5] 5 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.419763+0000 47'13 47:34 [6] 6 [6] 6 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.293914+0000 47'13 47:34 [6] 6 [6] 6 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-02-09T22:22:12.881669+0000 47'39 47:59 [4] 4 [4] 4 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-02-09T22:22:11.725070+0000 47'52 47:74 [3] 3 [3] 3 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-02-09T22:22:12.780970+0000 47'52 47:72 [4] 4 [4] 4 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.725062+0000 47'13 47:38 [0] 0 [0] 0 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:22:11.900033+0000 0'0 47:20 [3] 3 [3] 3 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.162 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.271480+0000 47'13 47:42 [5] 5 [5] 5 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.900728+0000 47'13 47:30 [0] 0 [0] 0 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.271399+0000 47'13 47:41 [7] 7 [7] 7 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.361155+0000 47'13 47:32 [7] 7 [7] 7 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.781341+0000 47'13 47:30 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.419724+0000 47'13 47:31 [6] 6 [6] 6 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.394669+0000 47'13 47:32 [5] 5 [5] 5 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.900141+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 2 0 0 0 0 94371840 0 0 23 0 23 active+clean 2024-02-09T22:22:11.900021+0000 47'23 47:41 [0] 0 [0] 0 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:12.881668+0000 47'13 47:29 [4] 4 [4] 4 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:22:12.755682+0000 0'0 47:17 [2] 2 [2] 2 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-02-09T22:22:12.781199+0000 47'39 47:55 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:22:12.419981+0000 0'0 47:17 [6] 6 [6] 6 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.163 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.900570+0000 47'13 47:31 [0] 0 [0] 0 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:22:12.394454+0000 0'0 47:18 [5] 5 [5] 5 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.725022+0000 47'13 47:31 [3] 3 [3] 3 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-09T22:22:12.881691+0000 0'0 47:16 [1] 1 [1] 1 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-09T22:22:11.900263+0000 47'13 47:31 [0] 0 [0] 0 0'0 2024-02-09T22:22:10.730803+0000 0'0 2024-02-09T22:22:10.730803+0000 0 0 periodic scrub scheduled @ 2024-02-10T22:22:10.730803+0000 0 0 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-09T22:23:20.164 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:20.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-02-09T22:23:20.280 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:23:20.280 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_recovery_max_active": "0" 2024-02-09T22:23:20.280 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:23:20.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-02-09T22:23:20.388 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:23:20.388 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_recovery_max_active_hdd": "3" 2024-02-09T22:23:20.388 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:23:20.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-02-09T22:23:20.559 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-02-09T22:23:20.559 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_recovery_max_active_ssd": "10" 2024-02-09T22:23:20.559 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-02-09T22:23:20.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-02-09T22:23:20.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:20.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:20.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:20.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:20.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:20.939 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:20.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-02-09T22:23:20.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:22.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-02-09T22:23:22.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-02-09T22:23:22.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-02-09T22:23:22.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:22.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:22.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:22.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:22.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:23.225 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:23.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:23.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:25.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-02-09T22:23:25.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-02-09T22:23:25.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-02-09T22:23:25.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:25.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:25.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:25.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:25.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:25.511 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:25.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:25.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:27.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-02-09T22:23:27.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-02-09T22:23:27.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-02-09T22:23:27.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:27.795 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:27.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:27.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:29.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 3 -eq 10 2024-02-09T22:23:29.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 3 + 1 2024-02-09T22:23:29.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=4 2024-02-09T22:23:29.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:29.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:29.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:29.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:29.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:30.078 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:30.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:30.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:32.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 4 -eq 10 2024-02-09T22:23:32.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 4 + 1 2024-02-09T22:23:32.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=5 2024-02-09T22:23:32.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:32.094 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:32.094 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:32.094 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:32.094 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:32.359 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:32.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:32.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:34.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 5 -eq 10 2024-02-09T22:23:34.372 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 5 + 1 2024-02-09T22:23:34.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=6 2024-02-09T22:23:34.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:34.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:34.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:34.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:34.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:34.643 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:34.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:34.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:36.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 6 -eq 10 2024-02-09T22:23:36.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 6 + 1 2024-02-09T22:23:36.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=7 2024-02-09T22:23:36.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:36.660 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:36.660 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:36.660 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:36.660 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:36.928 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:36.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:36.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 7 -eq 10 2024-02-09T22:23:38.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 7 + 1 2024-02-09T22:23:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=8 2024-02-09T22:23:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:38.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:38.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:38.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:38.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:39.211 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:39.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:39.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:41.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 8 -eq 10 2024-02-09T22:23:41.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 8 + 1 2024-02-09T22:23:41.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=9 2024-02-09T22:23:41.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:41.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:41.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:41.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:41.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:41.494 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:41.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:41.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:43.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 9 -eq 10 2024-02-09T22:23:43.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 9 + 1 2024-02-09T22:23:43.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=10 2024-02-09T22:23:43.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-09T22:23:43.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-09T22:23:43.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-09T22:23:43.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-09T22:23:43.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-09T22:23:43.776 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-02-09T22:23:43.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-09T22:23:43.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-09T22:23:45.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 10 -eq 10 2024-02-09T22:23:45.789 INFO:tasks.workunit.client.0.smithi134.stdout:Not enough recovery started simultaneously 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:293: TEST_recovery_scrub_2: echo 'Not enough recovery started simultaneously' 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:294: TEST_recovery_scrub_2: return 1 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: return 1 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 1 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-09T22:23:45.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-09T22:23:45.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-09T22:23:45.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-09T22:23:45.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-09T22:23:45.905 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-09T22:23:45.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-09T22:23:45.906 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-09T22:23:45.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-09T22:23:45.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-09T22:23:45.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-09T22:23:45.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:23:45.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-09T22:23:45.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-09T22:23:45.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:23:45.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:23:45.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-02-09T22:23:45.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-09T22:23:45.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-09T22:23:45.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-recovery-scrub/mgr.x.log td/osd-recovery-scrub/mon.a.log td/osd-recovery-scrub/osd.0.log td/osd-recovery-scrub/osd.1.log td/osd-recovery-scrub/osd.2.log td/osd-recovery-scrub/osd.3.log td/osd-recovery-scrub/osd.4.log td/osd-recovery-scrub/osd.5.log td/osd-recovery-scrub/osd.6.log td/osd-recovery-scrub/osd.7.log /home/ubuntu/cephtest/archive/log 2024-02-09T22:23:45.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-09T22:23:45.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-09T22:23:45.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-09T22:23:45.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44749 2024-02-09T22:23:45.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44749 2024-02-09T22:23:45.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-09T22:23:45.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-09T22:23:45.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-02-09T22:23:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:23:45.924 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-02-09T22:23:45.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-09T22:23:46.194 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh' 2024-02-09T22:23:46.421 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9d1ed95a945c402e858ce17e644fe5ba Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh' 2024-02-09T22:23:46.424 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-09T22:23:46.433 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-09T22:23:46.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T22:23:46.502 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-09T22:23:46.510 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-09T22:23:46.510 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T22:23:46.529 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-02-09T22:23:46.533 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T22:23:46.533 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-02-09T22:23:46.533 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 -283us[ -264us] +/- 89ms 2024-02-09T22:23:46.534 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1740us[-1720us] +/- 60ms 2024-02-09T22:23:46.534 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +2750us[+2769us] +/- 84ms 2024-02-09T22:23:46.534 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:23:46.535 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-09T22:23:46.543 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-09T22:23:46.544 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-09T22:23:46.553 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T22:23:46.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T22:23:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:23:46.666 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-09T22:23:46.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-09T22:23:46.683 INFO:teuthology.task.internal:Duration was 1036.352482 seconds 2024-02-09T22:23:46.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-09T22:23:46.692 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-09T22:23:46.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T22:23:46.749 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T22:23:47.015 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-09T22:23:47.015 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-02-09T22:23:47.016 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T22:23:47.081 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-09T22:23:47.081 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T22:23:47.150 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-09T22:23:47.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T22:23:47.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-09T22:23:47.291 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-09T22:23:47.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T22:23:47.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-09T22:23:47.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:23:47.384 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-02-09T22:23:47.411 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T22:23:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:23:47.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-09T22:23:47.475 INFO:teuthology.task.internal:Transferring archived files... 2024-02-09T22:23:47.476 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381/remote/smithi134 2024-02-09T22:23:47.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T22:23:49.223 INFO:teuthology.task.internal:Removing archive directory... 2024-02-09T22:23:49.223 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T22:23:49.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-09T22:23:49.289 INFO:teuthology.task.internal:Not uploading archives. 2024-02-09T22:23:49.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-09T22:23:49.299 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-09T22:23:49.299 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T22:23:49.333 INFO:teuthology.orchestra.run.smithi134.stdout: 265563 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 9 22:23 /home/ubuntu/cephtest 2024-02-09T22:23:49.334 INFO:teuthology.orchestra.run.smithi134.stdout: 403053 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 9 22:13 /home/ubuntu/cephtest/mnt.0 2024-02-09T22:23:49.334 INFO:teuthology.orchestra.run.smithi134.stdout: 403054 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 9 22:16 /home/ubuntu/cephtest/mnt.0/client.0 2024-02-09T22:23:49.334 INFO:teuthology.orchestra.run.smithi134.stdout: 403095 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 9 22:23 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-09T22:23:49.334 INFO:teuthology.orchestra.run.smithi134.stdout: 417135 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 22:23 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-02-09T22:23:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:23:49.335 INFO:teuthology.orchestra.run.smithi134.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-09T22:23:49.336 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-09T22:23:49.341 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-09T22:23:49.351 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-09T22:23:49.361 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi134 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-09T22:23:49.361 INFO:teuthology.nuke:Checking targets against current locks 2024-02-09T22:23:49.388 DEBUG:teuthology.nuke:shortname: smithi134 2024-02-09T22:23:49.389 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T22:23:49.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:36-rados-main-distro-default-smithi/7554381', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 21:58:38.728637', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBPPP9QvWd6JLIkaxOmMoCvhRQXYAPEWnAsQ1wjtiF/mqHI8zoRxfvUWu53BaX+tgKxG3Jt9aeNOXXt6sNf+3Q='} 2024-02-09T22:23:49.462 INFO:teuthology.orchestra.console.smithi134:Power off 2024-02-09T22:23:49.462 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T22:23:49.487 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-02-09T22:23:49.488 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:23:49.514 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-02-09T22:23:53.515 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:23:53.626 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-02-09T22:23:57.629 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:23:57.739 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-02-09T22:24:01.743 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T22:24:01.854 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-02-09T22:24:01.955 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-02-09T22:24:02.060 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 1036.3524820804596 failure_reason: 'Command failed (workunit test scrub/osd-recovery-scrub.sh) on smithi134 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9d1ed95a945c402e858ce17e644fe5ba status: fail success: false 2024-02-09T22:24:02.061 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T22:24:02.141 INFO:teuthology.run:FAIL